{"id":"ae6a4cba-e8e7-4f4c-966f-6210c34d5e0c","arxiv_id":"2412.06292","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ZeroKey detects 3D keypoints on unseen object categories by prompting the Molmo vision-language model on multiple rendered views and aggregating the back-projected points, with no 3D annotations required.","lead":"ZeroKey finds and names important points on 3D shapes by asking a vision-language AI to point at them across several rendered views, then combining those 2D answers into 3D locations. It needs no 3D keypoint training data, which could remove a costly annotation bottleneck in 3D shape analysis.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot generality rests on Molmo's 2D point accuracy on unseen categories, but only three hand-picked categories are evaluated and the 2D localization error is never isolated.","rationale":"The reader's weakest assumption (Molmo's 2D localization on rendered, unseen shapes) is genuinely load-bearing: if Molmo's point outputs are biased or inconsistent, no amount of back-projection or clustering can recover the correct semantic 3D point. I agree with that identification. I would extend it in two ways: the evaluation does not test the claimed arbitrary-category generality at all, and the manually curated prompts mean the automatic naming stage is never quantitatively validated. These are not internal inconsistencies; the pipeline is coherent and the reported results on the three tested categories give some evidence that the approach can work. However, the central claim is broader than the evidence. A check on all sixteen categories, with 2D localization accuracy separated from 3D aggregation, would settle whether the missing generality is real. The reader's CONDITIONAL verdict is appropriate; my concern strengthens the condition rather than changing the verdict.","tokens_in":18156,"tokens_out":8469,"duration_ms":96781,"concrete_test":"Render all 16 KeypointNet categories with the same 26 views, project each ground-truth keypoint to 2D, and prompt Molmo with the exact labels from Supplementary Table D. Compute per-category 2D PCK at 1% and 2% of the image diagonal, then run the full ZeroKey pipeline and report per-category 3D IoU. If the 13 held-out categories are substantially worse than airplane/chair/table, or if 2D PCK is low on any category, the arbitrary-category claim is unsupported; if performance is uniform, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The pipeline uses Eq. 1's Molmo 2D point as the only semantic signal; back-projection (Eqs. 4/6) and HDBSCAN clustering aggregate that signal but cannot correct it if the 2D point is wrong. The paper claims arbitrary-category generality, yet the quantitative evaluation covers only three of the sixteen KeypointNet categories (§6.1), and the prompts used are manually written from the ground-truth keypoint IDs (Supplementary D). No per-category IoU, no per-view 2D localization accuracy, and no error bars are reported. If Molmo's point-level outputs degrade on less common renderings (e.g., bathtub inner corners, helmet visor bottom) or on ambiguous names such as \"corner of the table\" (which the paper itself shows yields multiple plausible points), that error propagates directly into the 3D result and would be hidden by the aggregate averages. The manual prompt selection also conflates the proposed automatic keypoint naming step (§4.1) with externally supplied names, so the \"extracting and naming\" claim is not independently measured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ZeroKey, a zero-shot 3D keypoint detection pipeline that first uses GPT-4o to generate a list of salient keypoint names from rendered views, then prompts Molmo to localize each named keypoint in multiple rendered views, back-projects these 2D detections onto the mesh, and aggregates them via patch refinement and HDBSCAN clustering. The method is evaluated on three KeypointNet categories (airplane, chair, table) with IoU at varying geodesic thresholds, along with ablations over prompt specificity, number of views, and clustering, plus additional applications to Schelling points and point describability.","tokens_in":18244,"tokens_out":4547,"duration_ms":43201,"significance":"If the full pipeline works as claimed, this would be an interesting demonstration that pixel-level pretraining in MLLMs transfers to 3D keypoint detection without 3D annotations. The paper's strengths include a simple, reproducible pipeline built on frozen models, a clear ablation showing the importance of point-specific prompts and HDBSCAN aggregation, and public code. The performance gap over the CLIP-DINOiser and GPT-4o baselines is substantial. However, the evaluation as it stands does not yet support the full 'extracting and naming' generality claim: the reported numbers use manually supplied names derived from ground-truth keypoint IDs, and only three classes are tested.","major_comments":[{"comment":"The evaluation protocol manually attaches text descriptions to each ground-truth keypoint ID in KeypointNet and uses those descriptions as prompts to Molmo. This measures localization of externally specified keypoints, not the automatic extraction and naming step proposed in Section 4.1. The abstract's claim of 'extracting and naming salient keypoints' is therefore not supported by the quantitative results. A proper evaluation would run the full pipeline (GPT-4o candidate generation followed by Molmo localization) and measure how well the resulting keypoint set matches the ground truth, without constructing prompts from the ground-truth labels. This is load-bearing for the zero-shot claim.","section":"Section 6.1, Supplementary D"},{"comment":"The paper reports results only for airplane, chair, and table, three of the sixteen KeypointNet categories, and only as aggregate IoU curves with no per-category breakdown or variance. Given that the paper claims arbitrary-category generality and dedicates Supplementary D to all sixteen categories, the absence of per-category numbers is a significant omission; categories with many ambiguous or duplicated descriptions (e.g., bathtub, vessel) may behave very differently. At minimum, report per-category IoU and standard deviations across shapes, and ideally include at least one difficult category. The abstract's 'competitive performance compared to supervised methods' should also be qualified, since the supplementary table shows the method below the best few-shot methods (B2-3D, FSKD) at all thresholds.","section":"Section 6.1, Table 1, Supplementary Table 2"},{"comment":"The clustering and matching details needed to interpret the IoU numbers are underspecified. The paper does not state how HDBSCAN's minPts k and the patch size h are chosen, whether these are fixed across categories, or how predicted clusters are matched to ground-truth keypoints. Figure 4 shows that the number of detected points for a single prompt ('corner of the table') grows with the number of views, so the matching rule is not trivial; without it, the reported IoU cannot be reproduced. Please specify the exact matching protocol and report sensitivity to h, k, and M.","section":"Section 4.3, Eq. (7)"},{"comment":"The entire pipeline's upper bound is Molmo's 2D point localization accuracy on rendered synthetic views of unseen categories, but this quantity is never isolated. A back-projected 2D error propagates directly into the 3D estimate; clustering cannot correct a consistently wrong point. The paper should report per-view 2D localization error against projected ground-truth keypoints for the evaluated categories, and ideally on unseen categories, to establish where errors originate and whether the zero-shot claim holds under less favorable renderings.","section":"Section 4.2, Eq. (1), Section 4.3"}],"minor_comments":[{"comment":"The sentence 'Qualitative results are shown in Table 1' should be 'Quantitative results are shown in Table 1', since Table 1 contains numeric IoU values.","section":"Section 6.3"},{"comment":"The sentence 'Qualitative results are provided in Table 2' should likewise be 'Quantitative results are provided in Table 2'.","section":"Supplementary C"},{"comment":"The claim that 6 views achieve 80% of the full performance is not backed by axes or exact values in Figure 8; please state the number in the text and label the figure axes.","section":"Section 6.4"},{"comment":"The caption says that as the number of views increases, ZeroKey detects more keypoints for 'corner of the table'; this ambiguity is central to the evaluation, so clarify whether the evaluation prompts also yield multiple clusters and how these are counted in the IoU metric.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central idea is sound and the writing is clear. The main risk is not circularity (Molmo and GPT-4o are externally pretrained and no parameters are tuned on KeypointNet), but that the reported evaluation uses human-supplied names and only three classes, so the stated zero-shot generality claim is broader than the evidence. The requested additions—end-to-end evaluation with the automatic naming step, per-category results, and a 2D error analysis—are feasible and within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Rough take: this is the first paper I know that gets 3D keypoints from an open-world text prompt using a point-level MLLM, and the numbers show it works. The pipeline is simple — render views, ask Molmo for a 2D point per keypoint, back-project, cluster with HDBSCAN — but the components fit together and the ablations actually support the design choices. The comparison against CLIP-DINOiser and GPT-4o is fair and the margin is large. I also liked the Schelling point experiment: it is a clever way to show that the model's notion of salience lines up with human agreement.\n\nNow the soft spots. The quantitative evaluation is three KeypointNet classes out of sixteen, with no per-category breakdown and no error bars. The full table in the supplement shows the method trailing B2-3D and a few supervised detectors by a clear margin, so 'competitive' is doing a lot of work. More importantly, the prompts are manually written from the ground-truth keypoint IDs in the supplement, so the automatic naming step (Sec. 4.1) is never measured on its own. That weakens the 'without any ground truth labels or supervision' claim. The stress-test note is right that Molmo's 2D point is the only semantic signal; if it degrades on unseen categories or ambiguous names, back-projection and clustering can't fix that, and the paper gives no per-view 2D accuracy to rule that out. I'd also want to see the hyperparameters (patch size h, minPts k, number of views M) specified somewhere; right now they're absent.\n\nNone of this is fatal. The central idea is plausible, the baselines are reasonable, and the method is not circular — no parameter is fitted to KeypointNet. It's a solid contribution that would benefit from a broader evaluation and a more careful wording of the zero-shot claim.\n\nWho is it for? Anyone working on zero-shot 3D understanding or on lifting 2D foundation-model signals to 3D. It deserves a serious referee; I'd send it to review with the request for more categories, error bars, and a direct evaluation of the naming and 2D-localization stages.","headline":"First credible zero-shot 3D keypoint detector built on Molmo's point-level MLLM abilities, but the evaluation on three categories is too thin to support all the claims; still, the core idea is reproducible and worth reviewing.","tokens_in":18913,"tokens_out":1997,"would_cite":true,"duration_ms":20004,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ZeroKey detects named 3D keypoints on arbitrary object categories with no 3D keypoint annotations, by asking a point-aware multimodal language model to localize parts in rendered views and then aggregating the back-projected points.","keywords":["zero-shot 3D keypoint detection","multimodal large language models","point-level reasoning","multi-view aggregation","back-projection","HDBSCAN clustering","keypoint naming","Molmo"],"falsifier":"Test ZeroKey on a shape category that does not appear in Molmo's pixel-annotation dataset (for example, a watering can) and measure its geodesic IoU against a small human-annotated keypoint set; if at a 0.05 threshold the IoU is no better than the RedCircle random baseline, the localization signal is not genuinely zero-shot.","tokens_in":17856,"feed_emoji":"📍","tokens_out":6660,"duration_ms":61084,"temperature":0.7,"pith_summary":"The paper tries to establish that 3D keypoint detection can be done without any 3D keypoint annotations or training. It shows that the point-level localization ability of a multimodal language model (Molmo), trained on pixel-level annotations, transfers to rendered views of 3D shapes: ask it to 'point to the nose', back-project its 2D answers, and cluster them across views. On the KeypointNet benchmark the method matches or beats several supervised and few-shot detectors, and it works on arbitrary categories with no per-category adaptation. If the claim holds, 3D shape understanding no longer needs a supervised keypoint pipeline: any named part becomes an instant query.","feed_headline":"Point to the nose: a language model finds 3D keypoints","feed_subtitle":"ZeroKey renders a shape, asks Molmo to localize named parts per view, and clusters back-projections to rival supervised detectors.","key_machinery":"The load-bearing component is Molmo, an open multimodal language model trained with a dedicated pixel-level annotation task that lets it answer 'Point to {query} in this image' with a 2D coordinate. The method wraps this with three mechanisms: GPT-4o generates the candidate keypoint names; each of 26 rendered views is prompted with the name and the resulting 2D points are back-projected along cached depth rays using a small patch to stabilize sharp intersections; HDBSCAN then clusters the multi-view 3D candidates, returning only consistent keypoints and filtering outliers.","core_discovery":"The central discovery is that pixel-level annotation data used to train recent MLLMs carries usable 3D keypoint knowledge: without any ground-truth 3D labels, ZeroKey extracts and names salient keypoints on a 3D model by prompting Molmo to point at the keypoint in each rendered view, back-projecting the 2D points into 3D via cached per-pixel depth, and clustering across views with HDBSCAN to suppress noise. On KeypointNet (airplane, chair, table), the zero-shot method achieves an IoU of 54.64% at a geodesic threshold of 0.10, clearly above the text-to-point baselines (CLIP-DINOiser: 25.56, RedCircle: 18.50, GPT-4o: 20.73) and within the range of supervised and few-shot methods.","pith_inferences":["The 'zero-shot' claim is scoped to 3D annotations: Molmo's 2D localization was trained on pixel-annotated images, so the true test of generalization is whether that 2D knowledge transfers to synthetic renderings and categories absent from its annotation corpus.","The cluster density from HDBSCAN could be reused as a confidence measure for whether a named part exists on a given shape, giving the pipeline a built-in 'not present' signal that the paper does not exploit.","The method could be inverted: instead of asking GPT-4o for names, let Molmo describe a marked point and then re-find it, a self-supervised loop that the paper's describability experiments hint at but do not develop into a full consistency score.","Because candidate names come from a language model, the ceiling of performance is set by the naming step; using a part taxonomy or image-derived description rather than free-form names would likely reduce ambiguous prompts like 'corner of the table'."],"forward_implications":["Keypoint detection becomes a text query: any part that can be named can be localized on any shape, with no per-category training set.","The same multi-view lifting can produce pseudo-labels to bootstrap supervised 3D keypoint datasets on new categories.","The Schelling points study shows the recovered points align with human-chosen salient points, linking language describability to annotation agreement.","The method establishes a baseline for zero-shot 3D keypoint detection that future vision-language approaches can be measured against."],"supporting_citations":[{"why":"Molmo and PixMo: supplies the point-level localization backbone that makes 2D keypoint answering possible.","marker":"[11]"},{"why":"GPT-4: generates the candidate keypoint names and serves as an ablated image-only MLLM baseline.","marker":"[32]"},{"why":"KeypointNet: provides the ground-truth keypoints and dataset used for evaluation.","marker":"[50]"},{"why":"UKPGAN: supplies the evaluation protocol (IoU with geodesic distance thresholds) and a supervised comparison.","marker":"[51]"},{"why":"CLIP-DINOiser: a zero-shot text-driven localization method used as the main 2D-to-3D baseline to beat.","marker":"[46]"},{"why":"Red Circle: a simple text-driven point baseline lifted to 3D for comparison.","marker":"[40]"},{"why":"Schelling points: provides the human-saliency distributions used for the describability and consistency analysis.","marker":"[9]"},{"why":"B2-3D: a few-shot back-projected baseline that supplies the closest supervised comparison at matching view counts.","marker":"[45]"}],"fun_headline_variants":["ZeroKey: Zero-shot 3D keypoints via language model pointing","No labels? No problem: LLM knows 3D keypoints","Point to the nose: LLM teaches zero-shot 3D keypoints","Ask an LLM to point: zero-shot 3D keypoint detection","ZeroKey: zero-shot 3D keypoints from MLLM annotations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline's accuracy is inherited wholesale from Molmo: if its 2D point answers are wrong or inconsistent on rendered views of an unseen shape, the back-projection and clustering cannot recover, and the zero-shot result fails.","fun_headline_variants_meta":{"raw":{"variants":["ZeroKey: Zero-shot 3D keypoints via language model pointing","No labels? No problem: LLM knows 3D keypoints","Point to the nose: LLM teaches zero-shot 3D keypoints","Ask an LLM to point: zero-shot 3D keypoint detection","ZeroKey: zero-shot 3D keypoints from MLLM annotations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001147,"raw_usage":{"total_tokens":4749,"prompt_tokens":926,"completion_tokens":3823,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":3724}},"tokens_in":542,"tokens_out":3823,"duration_ms":23080,"temperature":1.0,"reasoning_tokens":3724,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:48:36.122851+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Test ZeroKey on a shape category that does not appear in Molmo's pixel-annotation dataset (for example, a watering can) and measure its geodesic IoU against a small human-annotated keypoint set; if at a 0.05 threshold the IoU is no better than the RedCircle random baseline, the localization signal is not genuinely zero-shot.","supporting_citations":[{"cited_title":"Gpt-4 technical report, 2023","cited_arxiv_id":null,"evidence_quote":"GPT-4: generates the candidate keypoint names and serves as an ablated image-only MLLM baseline."},{"cited_title":"Ukpgan: A general self-supervised keypoint detector","cited_arxiv_id":null,"evidence_quote":"UKPGAN: supplies the evaluation protocol (IoU with geodesic distance thresholds) and a supervised comparison."},{"cited_title":"Clip-dinoiser: Teaching clip a few dino tricks for open- 10 vocabulary semantic segmentation","cited_arxiv_id":null,"evidence_quote":"CLIP-DINOiser: a zero-shot text-driven localization method used as the main 2D-to-3D baseline to beat."},{"cited_title":"What does clip know about a red circle? vi- sual prompt engineering for vlms","cited_arxiv_id":null,"evidence_quote":"Red Circle: a simple text-driven point baseline lifted to 3D for comparison."},{"cited_title":"Schelling points on 3d surface meshes","cited_arxiv_id":null,"evidence_quote":"Schelling points: provides the human-saliency distributions used for the describability and consistency analysis."},{"cited_title":"Back to 3d: Few-shot 3d keypoint detection with back-projected 2d features","cited_arxiv_id":null,"evidence_quote":"B2-3D: a few-shot back-projected baseline that supplies the closest supervised comparison at matching view counts."}],"review_version":1}