Pith. sign in

REVIEW 3 major objections 7 minor 78 references

Zero-Shot 3D Visual Grounding from Vision-Language Models

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SeeGround localizes 3D objects with a 2D vision-language model and no 3D-specific training, outperforming prior zero-shot methods by 7.7% on ScanRefer and 7.1% on Nr3D.

desk verdict Solid, honestly ablated method for zero-shot 3DVG; the headline ScanRefer gain is confounded by VLM choice and needs a controlled re-run. read the letter →

arxiv 2505.22429 v1 pith:7KMO7XYD submitted 2025-05-28 cs.CV cs.RO

Rong Li , Shijie Li , Lingdong Kong , Xulei Yang , Junwei Liang This is my paper
classification cs.CVcs.RO
keywords 3Dvisualgroundingzero-shotlearningvision-languagemodelquery-alignedrenderingdepth-awarepromptingobjectlookuptableScanReferNr3D
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that 3D visual grounding can be done in a zero-shot, training-free way by feeding a 2D vision-language model a hybrid representation: an image rendered from a query-chosen viewpoint, paired with structured text describing the 3D positions of detected objects. This matters because conventional 3D grounding models depend on dense 3D annotations and fixed object categories, which limit them in open-world settings. SeeGround's accuracy comes from aligning the rendered view with the spatial text through depth-aware visual prompts, so the VLM can read fine-grained cues such as color, texture, shape, and orientation. The paper reports 44.1% overall accuracy on ScanRefer and 46.1% on Nr3D, surpassing prior zero-shot baselines by 7.7 and 7.1 points, respectively, and approaching fully supervised models.

What carries the argument

The load-bearing mechanism is the hybrid scene representation: a query-aligned rendered image combined with structured spatial text drawn from an Object Lookup Table (OLT) built by an open-vocabulary 3D detector. Two modules refine this representation: the Perspective Adaptation Module selects the camera pose by facing the parsed anchor object (or the centroid of candidates when no anchor is confident), and the Fusion Alignment Module projects each object's visible 3D points onto the image, filters occluded points by depth, and overlays a visual prompt at each visible object's projected center. This explicit alignment between 2D visual cues and 3D spatial semantics is what lets the 2D VLM distinguish objects that differ only by texture, shape, viewpoint, orientation, state, or order.

What would settle it

A concrete test: filter the ScanRefer evaluation to queries whose target object is not detected by Mask3D (no prediction overlapping the ground-truth box at IoU > 0.25) and measure SeeGround's accuracy on that subset. The design of the Object Lookup Table implies the accuracy should be 0, since the VLM can only choose among listed IDs; any nonzero accuracy, or a failure of accuracy to rise toward the 59.5% seen with ground-truth boxes, would contradict the claimed mechanism.

Watch

Extended reading notes

Core claim

SeeGround's central claim is that a pretrained 2D VLM, given a query-aligned rendered image and spatially enriched text, can localize a referred object in a 3D scene without any 3D-specific training. The pipeline parses the query into a target and an anchor object, selects a viewpoint facing the anchor, renders the scene from that viewpoint, projects the visible object boxes onto the image as prompts, and asks the VLM to output the target's ID. That ID is then looked up in the Object Lookup Table, which the open-vocabulary 3D detector populated with boxes and labels. The authors report gains of 7.7% on ScanRefer and 7.1% on Nr3D over prior zero-shot methods, with particular robustness on view-dependent and hard queries.

Load-bearing premise

The load-bearing premise is that the target object and whatever anchor objects the query mentions are present in the Object Lookup Table produced by the off-the-shelf 3D detector, with correct 3D boxes; if the detector misses the referred object, SeeGround has no ID to output and must fail.

Editorial extensions

If this is right

  • On ScanRefer validation, SeeGround improves overall accuracy to 44.1% at Acc@0.25 from 36.4% of the prior zero-shot baseline, while on the unique-object split it reaches 75.7%.
  • On Nr3D, the 46.1% overall accuracy beats ZSVG3D by 7.1 points, with the largest gains on view-dependent queries.
  • Replacing the Mask3D detector with ground-truth boxes raises accuracy to 59.5%, showing that remaining error is dominated by detector coverage and label accuracy rather than VLM reasoning.
  • The robust-incomplete-query experiment shows that when anchor objects are omitted from the text, SeeGround still grounds the target using visual context, whereas an LLM-only pipeline fails.
  • Because the pipeline is training-free, the same OLT and rendering stack can be reused for new queries without any per-scene learning, and new open-vocabulary detectors can be swapped in.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the detector ceiling is the main bottleneck, then improving open-vocabulary 3D detection recall should transfer almost linearly into grounding accuracy; this is testable by pairing SeeGround with a stronger detector and checking whether the ScanRefer number moves toward the 59.5% ground-truth ceiling.
  • The framework's indoor assumptions — a single room center, closed-room cropping, and a fixed top-down camera height — suggest it will need explicit handling of unbounded outdoor scenes, where there is no natural 'scene center' and occlusion patterns differ.
  • The same hybrid rendering-and-prompting recipe could be pointed at other language-conditioned 3D tasks such as referring segmentation or spatial question answering, since the OLT plus query-aligned render already carries the necessary information.
  • The 8.4-point gap between text-only (37.7%) and full (46.1%) configurations is an upper bound on how much pure visual reasoning contributes; a VLM with stronger texture and state discrimination would be expected to narrow the remaining gap to the ground-truth ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. SeeGround is a training-free 3D visual grounding framework that converts 3D scenes into a hybrid representation of query-aligned rendered 2D images and structured spatial text, which is then fed to a 2D vision-language model (Qwen2-VL-72b). The method uses a Perspective Adaptation Module to choose a viewpoint relative to a parsed anchor object and a Fusion Alignment Module that projects visible 3D boxes into the image as visual prompts. On ScanRefer and Nr3D, the paper reports state-of-the-art zero-shot results, claiming improvements of 7.7% and 7.1% over previous zero-shot baselines, while honestly documenting that the off-the-shelf 3D detector (Mask3D) is the main bottleneck with a ground-truth-box ceiling of 59.5% on ScanRefer.

Significance. If the reported gains hold, SeeGround is a useful contribution to zero-shot 3DVG: it is training-free for the grounding module, uses an open-source VLM, and releases code. The paper's strengths include a clean overall pipeline, a clear error analysis (Fig. 7), and an honest detector-bottleneck table (Table 5) that shows the upper bound. The re-evaluation of ZSVG3D with the same Qwen2-VL on Nr3D (Fig. 5b) is a positive step toward controlled comparison. However, the headline ScanRefer comparison is confounded by VLM choice, and two other comparison/ablation issues (VLM-Grounder subset evaluation, and the PAM ablation in Table 3) weaken the central claims in the current form. These are fixable with additional experiments and presentation revisions, so the paper merits major revision rather than rejection.

major comments (3)
  1. [§4.2, Table 1, Fig. 5b] The headline +7.7% ScanRefer improvement over ZSVG3D is not a controlled comparison: SeeGround uses Qwen2-VL-72b while the reported ZSVG3D baseline uses GPT-4 turbo (Table 1). The paper re-evaluates ZSVG3D with Qwen2-VL only on Nr3D (Fig. 5b), not on ScanRefer. Since the entire pipeline's performance is mediated by VLM reasoning, the ScanRefer gain could be substantially due to the stronger language model. Please report ZSVG3D with Qwen2-VL-72b on the ScanRefer validation set (or the same VLM that SeeGround uses), and either update the claimed improvement or qualify it as VLM-dependent.
  2. [§4.3, Table 3] The ablation of the Perspective Adaptation Module (PAM) is confounded. The text claims that adding PAM improves accuracy from 43.3% (row (d)) to 45.0% (row (e)), but row (e) also removes the '3D Pos.' input (✗ in the 3D Pos column), so the gain cannot be uniquely attributed to PAM. The correct pairwise comparison would be (d) versus a configuration with 3D Pos+Layout+Texture+FAM+PAM. Although Table 4's viewpoint-strategy comparison supports the value of query-aligned rendering, the component-wise ablation as presented does not isolate PAM's contribution. Please revise the ablation design or state explicitly that PAM replaces raw 3D coordinates in that configuration.
  3. [§4.2, Table 1] The comparison with VLM-Grounder is not apples-to-apples. VLM-Grounder's results are marked with an asterisk indicating evaluation on 250 selected samples, and it reports 51.6/32.8 Acc@0.25/0.5 overall, which is higher than SeeGround's 44.1/39.4 on the full validation set. The abstract claims a 7.7% improvement over existing zero-shot baselines without excluding VLM-Grounder. Please clarify whether SeeGround achieves state-of-the-art on the full ScanRefer set (which would hold against ZSVG3D but not against VLM-Grounder's subset number), or report VLM-Grounder on the same evaluation protocol.
minor comments (7)
  1. [Table 4] There is a typo in the second row: 'Edege2Center' should be 'Edge2Center'.
  2. [§4.1] The implementation details specify the rendering resolution (1000x1000) and the top 0.3m exclusion, but the free parameters of the virtual camera (backward/upward shift, intrinsics, exclusion height) are not varied or justified. Please add a sensitivity analysis or state explicitly that these values are fixed and were chosen without test-set tuning.
  3. [§4.2, Table 1] The term 'zero-shot' should be qualified: Mask3D is a trained 3D instance segmentation model, so the method is training-free for the grounding module but not for the detector. A clarifying sentence in Section 4.1 would help avoid misunderstanding.
  4. [§3.1, Eq. (1)] The notation (bbox, sem) with a subscript N is not defined; please clarify that it denotes a set of N detected objects.
  5. [Table 5] The row labeled 'Ground Truth' appears under the SeeGround column; consider labeling it 'SeeGround + GT boxes' to avoid confusion with the method name in the header.
  6. [References] Reference [39] (arXiv:2412.04383) has the same title as this paper; please clarify whether it is a prior version of this work and how the present submission differs.
  7. [Figure 7] The error categories (Rel., Cls., View., Loc.) in the pie charts are not defined in the text; please add one-sentence definitions for each category.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the central results are held-out benchmark evaluations and the one self-citation is not load-bearing.

full rationale

SeeGround's derivation chain is self-contained: the pipeline runs an off-the-shelf detector (Mask3D), frozen 2D VLMs, hand-designed rendering and prompting, and is evaluated on held-out ScanRefer/Nr3D splits. No parameter is fit to the test-set targets, and Eq. (4) simply predicts an object ID from a prompted image and spatial text; the claimed gains are externally measured accuracies, not quantities reconstructed from the method's own inputs. The self-citation [39] supports only the generic statement that reasoning over 3D content can happen without 3D-specific training, and the paper's own experiments plus the ground-truth-box ceiling (Table 5: 44.1 with Mask3D vs 59.5 with GT boxes) independently substantiate that, so the citation is not load-bearing. The viewpoint strategy was selected by comparing hand-designed variants on the Nr3D validation set (Table 4), which is mild design tuning rather than a fitted parameter later renamed as a prediction. The ScanRefer comparison uses a different VLM than the baseline (Qwen2-VL-72b vs GPT-4 turbo), which is a legitimate experimental-confound concern, but that is a correctness/fairness issue, not circularity: the reported number remains an externally evaluated accuracy. No equation in the paper reduces a predicted quantity to its input by construction, and no load-bearing premise depends on an unverified self-citation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the 3D detector's completeness and label quality (grounded in Table 5), the 2D VLM's zero-shot abilities, and the fidelity of point-cloud rendering. No new physical entities are introduced. The hand-set rendering constants are listed as free parameters; they affect performance but are not fitted to the test distribution.

free parameters (3)
  • Virtual camera placement (backward and upward shift) = not reported
    Hand-set constants for the look-at-view-transform in Sec. 3.2; no sensitivity analysis is given.
  • Virtual camera intrinsics = 35mm focal length, principal point (30px, 20px)
    Fixed in the rendering pipeline (Fig. 2); hand-chosen.
  • Rendering exclusion height = 0.3m
    Sec. 4.1: 'excluding the top 0.3m to match closed-room settings'; arbitrary threshold.
assumptions (4)
  • domain assumption The 3D detector (Mask3D or OVIR-3D) returns an object set that includes the target and referable anchor objects with accurate 3D bounding boxes.
    Section 3.1 Eq. 1; if target is missing from OLT, the VLM cannot output its ID. Table 5 shows accuracy limited by detector (GT boxes 59.5% vs Mask3D 44.1%).
  • domain assumption The 2D VLM (Qwen2-VL-72B) can parse open-vocabulary queries into target/anchor pairs and can associate visual prompts with objects in rendered images.
    Sections 3.2 and 3.3; the entire pipeline depends on the VLM's zero-shot capabilities, which are not guaranteed for all scenes.
  • domain assumption Rendered point-cloud images preserve sufficient visual detail (color, texture, shape) for the VLM to discriminate objects.
    Section 3.1; the error analysis in Sec. 4.3 notes limited rendering quality hampers object discrimination.
  • standard math The pinhole camera projection and depth-buffer occlusion test correctly map 3D boxes to 2D prompts.
    Used in the Fusion Alignment Module; straightforward projection, no issue.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-Shot 3D Visual Grounding from Vision-Language Models." pith.science (2026). https://pith.science/paper/7KMO7XYD

@misc{pith2026250522429,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot 3D Visual Grounding from Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7KMO7XYD}},
  note         = {Machine review of arXiv:2505.22429}
}
read the original abstract

3D Visual Grounding (3DVG) seeks to locate target objects in 3D scenes using natural language descriptions, enabling downstream applications such as augmented reality and robotics. Existing approaches typically rely on labeled 3D data and predefined categories, limiting scalability to open-world settings. We present SeeGround, a zero-shot 3DVG framework that leverages 2D Vision-Language Models (VLMs) to bypass the need for 3D-specific training. To bridge the modality gap, we introduce a hybrid input format that pairs query-aligned rendered views with spatially enriched textual descriptions. Our framework incorporates two core components: a Perspective Adaptation Module that dynamically selects optimal viewpoints based on the query, and a Fusion Alignment Module that integrates visual and spatial signals to enhance localization precision. Extensive evaluations on ScanRefer and Nr3D confirm that SeeGround achieves substantial improvements over existing zero-shot baselines -- outperforming them by 7.7% and 7.1%, respectively -- and even rivals fully supervised alternatives, demonstrating strong generalization under challenging conditions.

Figures

Figures reproduced from arXiv: 2505.22429 by the authors.

Figure 1
Figure 1. Effectiveness of SeeGround: Unlike previous state-of-the-art methods, our approach aligns 2D visual cues – such as texture, shape, viewpoint, spatial position, orientation, state, and order – with 3D spatial language to enable fine-grained scene comprehension. Specifically, our method: (a) texture: detects the floral chair by leveraging distinctive color and texture patterns; (b) shape: identifies the couch through … view at source ↗
Figure 2
Figure 2. Overview of the SeeGround framework. A 2D-VLM first interprets the query, identifying the target (e.g., “laptop”) and an anchor (e.g., “chair with a floral pattern”). A dynamic viewpoint is selected based on the anchor’s position to render a query-aligned 2D image. Using the Object Lookup Table (OLT ), we retrieve 3D boxes, project visible ones, and apply visual prompts to reduce occlusion. The prompted image, spati… view at source ↗
Figure 3
Figure 3. Illustrative example of different perspective selection [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative Results. Rendered scenes with model pre￾dictions: correct objects in Green, incorrect in Orange. Key visual cues (e.g., color, texture, spatial relations) are underlined to illustrate the model’s reasoning. 38.3 42.3 48.2 54.5 33.40 40.10 41.10 49.20 29.6 3…
Figure 6
Figure 6. Figure 6: Robustness example: our method correctly identifies the [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 5
Figure 5. Figure 5: Ablation study on (a) different projection strategies (ours vs. ZSVG3D [32]), and (b) different language agents (GPT￾4 [35] vs. Qwen2-VL [36]). Qwen2-VL in place of GPT-4 [35] (Fig. 5b). Our method consistently outperforms ZSVG3D under the same VLM, confirming the effe…
Figure 7
Figure 7. Figure 7: Error distributions for the Text-Only method (a) and ours [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 62 canonical work pages

  1. [39]

    Seeground: See and ground for zero- shot open-vocabulary 3d visual grounding,

    R. Liet al., “Seeground: See and ground for zero- shot open-vocabulary 3d visual grounding,”arXiv preprint arXiv:2412.04383, 2024

    arXiv 2024

  2. [1]

    Scanrefer: 3d object localization in rgb-d scans using natural language,

    D. Z. Chenet al., “Scanrefer: 3d object localization in rgb-d scans using natural language,” inECCV, pp. 202–221, 2020

    work page 2020

  3. [2]

    RayDF: Neural Ray-surface Distance Fields with Multi-view Consistency

    Z. Liuet al., “Raydf: neural ray-surface distance fields with multi-view consistency,”arXiv preprint arXiv:2310.19629, 2023

    work page Pith review arXiv 2023

  4. [3]

    Deep view synthesis via self-consistent gen- erative network,

    Z. Liuet al., “Deep view synthesis via self-consistent gen- erative network,”IEEE Transactions on Multimedia, vol. 24, pp. 451–465, 2021

    work page 2021

  5. [4]

    Unleashing the potential of multi-modal foun- dation models and video diffusion for 4d dynamic physical scene simulation,

    Z. Liuet al., “Unleashing the potential of multi-modal foun- dation models and video diffusion for 4d dynamic physical scene simulation,”arXiv preprint arXiv:2411.14423, 2024

    arXiv 2024

  6. [5]

    SIR: Multi-view Inverse Rendering with Decomposable Shadow Under Indoor Intense Lighting

    X. Weiet al., “Sir: Multi-view inverse rendering with decomposable shadow for indoor scenes,”arXiv preprint arXiv:2402.06136, 2024

    work page Pith review arXiv 2024

  7. [6]

    An Examination of the Compositionality of Large Generative Vision-Language Models

    T. Maet al., “An examination of the compositionality of large generative vision-language models,”arXiv preprint arXiv:2308.10509, 2023

    work page Pith review arXiv 2023

  8. [7]

    Think global, act local: Dual-scale graph transformer for vision-and-language navigation,

    S. Chenet al., “Think global, act local: Dual-scale graph transformer for vision-and-language navigation,” inCVPR, pp. 16537–16547, 2022

    work page 2022

Show all 78 references
  1. [8]

    Assister: Assistive navigation via condi- tional instruction generation,

    Z. Huanget al., “Assister: Assistive navigation via condi- tional instruction generation,” inECCV, pp. 271–289, 2022

    2022

  2. [9]

    From cognition to precognition: A future- aware framework for social navigation,

    Z. Gonget al., “From cognition to precognition: A future- aware framework for social navigation,”arXiv preprint arXiv:2409.13244, 2024

    2024 arXiv

  3. [10]

    Clip2scene: Towards label-efficient 3d scene understanding by clip,

    R. Chenet al., “Clip2scene: Towards label-efficient 3d scene understanding by clip,” inCVPR, pp. 7020–7030, 2023

    2023

  4. [11]

    Robo3d: Towards robust and reliable 3d perception against corruptions,

    L. Konget al., “Robo3d: Towards robust and reliable 3d perception against corruptions,” inICCV, pp. 19994–20006, 2023

    2023

  5. [12]

    Rethinking range view representation for lidar segmentation,

    L. Konget al., “Rethinking range view representation for lidar segmentation,” inICCV, pp. 228–240, 2023

    2023

  6. [13]

    Xvo: Generalized visual odometry via cross- modal self-training,

    L. Laiet al., “Xvo: Generalized visual odometry via cross- modal self-training,” inICCV, pp. 10094–10105, 2023

    2023

  7. [14]

    Coarse3d: Class-prototypes for contrastive learning in weakly-supervised 3d point cloud segmentation,

    R. Liet al., “Coarse3d: Class-prototypes for contrastive learning in weakly-supervised 3d point cloud segmentation,” arXiv preprint arXiv:2210.01784, 2022

    2022 arXiv

  8. [15]

    Perception-aware multi-sensor fusion for 3d lidar semantic segmentation,

    Z. Zhuanget al., “Perception-aware multi-sensor fusion for 3d lidar semantic segmentation,” inICCV, pp. 16280–16290, 2021

    2021

  9. [16]

    Epmf: Efficient perception-aware multi- sensor fusion for 3d semantic segmentation,

    M. Tanet al., “Epmf: Efficient perception-aware multi- sensor fusion for 3d semantic segmentation,”TPAMI, vol. 46, no. 12, pp. 8258–8273, 2024

    2024

  10. [17]

    Tfnet: Exploiting temporal cues for fast and accurate lidar semantic segmentation,

    R. Liet al., “Tfnet: Exploiting temporal cues for fast and accurate lidar semantic segmentation,” inCVPR, pp. 4547– 4556, 2024

    2024

  11. [18]

    Robust 3d semantic occupancy prediction with calibration-free spatial transformation,

    Z. Zhuanget al., “Robust 3d semantic occupancy prediction with calibration-free spatial transformation,”arXiv preprint arXiv:2411.12177, 2024

    2024 arXiv

  12. [19]

    Dhp-mapping: A dense panoptic mapping sys- tem with hierarchical world representation and label opti- mization techniques,

    T. Huet al., “Dhp-mapping: A dense panoptic mapping sys- tem with hierarchical world representation and label opti- mization techniques,” inIROS, pp. 1101–1107, 2024

    2024

  13. [20]

    Multi-modal data-efficient 3d scene un- derstanding for autonomous drivin,

    L. Konget al., “Multi-modal data-efficient 3d scene un- derstanding for autonomous drivin,”TPAMI, vol. 47, no. 5, pp. 3748–3765, 2025

    2025

  14. [21]

    Dynamiccity: Large-scale 4d occu- pancy generation from dynamic scenes,

    H. Bianet al., “Dynamiccity: Large-scale 4d occu- pancy generation from dynamic scenes,”arXiv preprint arXiv:2410.18084, 2024

    2024

  15. [22]

    Calib3d: Calibrating model preferences for reliable 3d scene understanding,

    L. Konget al., “Calib3d: Calibrating model preferences for reliable 3d scene understanding,” inWACV, pp. 1965–1978, 2025

    1965

  16. [23]

    Bottom up top down detection transform- ers for language grounding in images and point clouds,

    A. Jainet al., “Bottom up top down detection transform- ers for language grounding in images and point clouds,” in ECCV, pp. 417–433, 2022. 7

    2022

  17. [24]

    3d-vista: Pre-trained transformer for 3d vision and text alignment,

    Z. Zhuet al., “3d-vista: Pre-trained transformer for 3d vision and text alignment,” inICCV, pp. 2911–2921, 2023

    2023

  18. [25]

    Eda: Explicit text-decoupling and dense align- ment for 3d visual grounding,

    Y . Wuet al., “Eda: Explicit text-decoupling and dense align- ment for 3d visual grounding,” inCVPR, pp. 19231–19242, 2023

    2023

  19. [26]

    3dvg-transformer: Relation modeling for vi- sual grounding on point clouds,

    L. Zhaoet al., “3dvg-transformer: Relation modeling for vi- sual grounding on point clouds,” inICCV, pp. 2928–2937, 2021

    2021

  20. [27]

    Instancerefer: Cooperative holistic under- standing for visual grounding on point clouds through in- stance multi-level contextual referring,

    Z. Yuanet al., “Instancerefer: Cooperative holistic under- standing for visual grounding on point clouds through in- stance multi-level contextual referring,” inICCV, pp. 1791– 1800, 2021

    2021

  21. [28]

    Multi-branch collaborative learning network for 3d visual grounding,

    Z. Qianet al., “Multi-branch collaborative learning network for 3d visual grounding,” inECCV, pp. 381–398, 2025

    2025

  22. [29]

    Semantickitti: A dataset for semantic scene understanding of lidar sequences,

    J. Behleyet al., “Semantickitti: A dataset for semantic scene understanding of lidar sequences,” inICCV, pp. 9297–9307, 2019

    2019

  23. [30]

    Scalability in perception for autonomous driv- ing: Waymo open dataset,

    P. Sunet al., “Scalability in perception for autonomous driv- ing: Waymo open dataset,” inCVPR, pp. 2446–2454, 2020

    2020

  24. [31]

    Panoptic nuscenes: A large-scale bench- mark for lidar panoptic segmentation and tracking,

    W. K. Fonget al., “Panoptic nuscenes: A large-scale bench- mark for lidar panoptic segmentation and tracking,”RA-L, vol. 7, pp. 3795–3802, 2022

    2022

  25. [32]

    Visual programming for zero-shot open- vocabulary 3d visual grounding,

    Z. Yuanet al., “Visual programming for zero-shot open- vocabulary 3d visual grounding,” inCVPR, pp. 20623– 20633, 2024

    2024

  26. [33]

    Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent,

    J. Yanget al., “Llm-grounder: Open-vocabulary 3d visual grounding with large language model as an agent,” inICRA, pp. 7694–7701, 2024

    2024

  27. [34]

    Training language models to fol- low instructions with human feedback,

    L. Ouyanget al., “Training language models to fol- low instructions with human feedback,”NeurIPS, vol. 35, pp. 27730–27744, 2022

    2022

  28. [35]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  29. [36]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wanget al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,”arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  30. [37]

    Cogvlm2: Visual language mod- els for image and video understanding,

    W. Honget al., “Cogvlm2: Visual language mod- els for image and video understanding,”arXiv preprint arXiv:2408.16500, 2024

    2024 arXiv

  31. [38]

    Sceneverse: Scaling 3d vision-language learn- ing for grounded scene understanding,

    B. Jiaet al., “Sceneverse: Scaling 3d vision-language learn- ing for grounded scene understanding,” inECCV, pp. 289– 310, 2025

    2025

  32. [40]

    Referit3d: Neural listeners for fine- grained 3d object identification in real-world scenes,

    P. Achlioptaset al., “Referit3d: Neural listeners for fine- grained 3d object identification in real-world scenes,” in ECCV, pp. 422–440, 2020

    2020

  33. [41]

    Viewrefer: Grasp the multi-view knowledge for 3d visual grounding,

    Z. Guoet al., “Viewrefer: Grasp the multi-view knowledge for 3d visual grounding,” inICCV, pp. 15372–15383, 2023

    2023

  34. [42]

    Multi-view transformer for 3d visual grounding,

    S. Huanget al., “Multi-view transformer for 3d visual grounding,” inCVPR, pp. 15524–15533, 2022

    2022

  35. [43]

    Look around and refer: 2d synthetic se- mantics knowledge distillation for 3d visual grounding,

    E. M. Bakret al., “Look around and refer: 2d synthetic se- mantics knowledge distillation for 3d visual grounding,” in NeurIPS, vol. 35, pp. 37146–37158, 2022

    2022

  36. [44]

    Sat: 2d semantics assisted training for 3d visual grounding,

    Z. Yanget al., “Sat: 2d semantics assisted training for 3d visual grounding,” inICCV, pp. 1856–1866, 2021

    2021

  37. [45]

    Four ways to improve verbo-visual fusion for dense 3d visual grounding,

    O. Unalet al., “Four ways to improve verbo-visual fusion for dense 3d visual grounding,” inECCV, pp. 196–213, 2025

    2025

  38. [46]

    Distilling coarse-to-fine semantic matching knowledge for weakly supervised 3d visual grounding,

    Z. Wanget al., “Distilling coarse-to-fine semantic matching knowledge for weakly supervised 3d visual grounding,” in ICCV, pp. 2662–2671, 2023

    2023

  39. [47]

    Unifying 3d vision-language understanding via promptable queries,

    Z. Zhuet al., “Unifying 3d vision-language understanding via promptable queries,” inECCV, pp. 188–206, 2024

    2024

  40. [48]

    Ovgaussian: Generalizable 3d gaus- sian segmentation with open vocabularies,

    R. Chenet al., “Ovgaussian: Generalizable 3d gaus- sian segmentation with open vocabularies,”arXiv preprint arXiv:2501.00326, 2025

    2025 arXiv

  41. [49]

    Multi-space alignments towards universal lidar segmentation,

    Y . Liuet al., “Multi-space alignments towards universal lidar segmentation,” inCVPR, pp. 14648–14661, 2024

    2024

  42. [50]

    Towards label-free scene understanding by vision foundation models,

    R. Chenet al., “Towards label-free scene understanding by vision foundation models,” inNeurIPS, pp. 75896–75910, 2023

    2023

  43. [51]

    Limoe: Mixture of lidar representation learners from automotive scenes,

    X. Xuet al., “Limoe: Mixture of lidar representation learners from automotive scenes,”arXiv preprint arXiv:2501.04004, 2025

    2025 arXiv

  44. [52]

    Geal: Generalizable 3d affordance learning with cross-modal consistency,

    D. Luet al., “Geal: Generalizable 3d affordance learning with cross-modal consistency,”arXiv preprint arXiv:2412.09511, 2025

    2025 arXiv

  45. [53]

    Openscene: 3d scene understanding with open vocabularies,

    S. Penget al., “Openscene: 3d scene understanding with open vocabularies,” inCVPR, pp. 815–824, 2023

    2023

  46. [54]

    Lerf: Language embedded radiance fields,

    J. Kerret al., “Lerf: Language embedded radiance fields,” in ICCV, pp. 19729–19739, 2023

    2023

  47. [55]

    Ovir-3d: Open-vocabulary 3d instance retrieval without training on 3d data,

    S. Luet al., “Ovir-3d: Open-vocabulary 3d instance retrieval without training on 3d data,” inCoRL, pp. 1610–1620, 2023

    2023

  48. [56]

    Agent3d-zero: An agent for zero-shot 3d understanding,

    S. Zhanget al., “Agent3d-zero: An agent for zero-shot 3d understanding,” inarXiv preprint arXiv:2403.11835, 2024

    2024 arXiv

  49. [57]

    Regionplc: Regional point-language con- trastive learning for open-world 3d scene understanding,

    J. Yanget al., “Regionplc: Regional point-language con- trastive learning for open-world 3d scene understanding,” in CVPR, pp. 19823–19832, 2024

    2024

  50. [58]

    Openmask3d: Open-vocabulary 3d instance segmentation,

    A. Takmazet al., “Openmask3d: Open-vocabulary 3d instance segmentation,”arXiv preprint arXiv:2306.13631, 2023

    2023 arXiv

  51. [59]

    Openins3d: Snap and lookup for 3d open- vocabulary instance segmentation,

    Z. Huanget al., “Openins3d: Snap and lookup for 3d open- vocabulary instance segmentation,” inECCV, pp. 169–185, 2025

    2025

  52. [60]

    Sai3d: Segment any instance in 3d scenes,

    Y . Yinet al., “Sai3d: Segment any instance in 3d scenes,” in CVPR, pp. 3292–3302, 2024

    2024

  53. [61]

    Lasermix for semi-supervised lidar semantic segmentation,

    L. Konget al., “Lasermix for semi-supervised lidar semantic segmentation,” inCVPR, pp. 21705–21715, 2023

    2023

  54. [62]

    Segment any point cloud sequences by distilling vision foundation models,

    Y . Liuet al., “Segment any point cloud sequences by distilling vision foundation models,” inNeurIPS, vol. 36, pp. 37193–37229, 2023

    2023

  55. [63]

    4d contrastive superflows are dense 3d repre- sentation learners,

    X. Xuet al., “4d contrastive superflows are dense 3d repre- sentation learners,” inECCV, pp. 58–80, 2024

    2024

  56. [64]

    Frnet: Frustum-range networks for scalable lidar segmentation,

    X. Xuet al., “Frnet: Frustum-range networks for scalable lidar segmentation,”TIP, vol. 34, pp. 2173–2186, 2025

    2025

  57. [65]

    Scene-llm: Extending language model for 3d visual understanding and reasoning,

    R. Fuet al., “Scene-llm: Extending language model for 3d visual understanding and reasoning,”arXiv preprint arXiv:2403.11401, 2024

    2024 arXiv

  58. [66]

    Uni3dl: A unified model for 3d and language understanding,

    X. Liet al., “Uni3dl: A unified model for 3d and language understanding,”arXiv preprint arXiv:2312.03026, 2023

    2023 arXiv

  59. [67]

    Conceptfusion: Open-set multi- modal 3d mapping,

    K. M. Jatavallabhulaet al., “Conceptfusion: Open-set multi- modal 3d mapping,”Robotics: Science and Systems, 2023. 8

    2023

  60. [68]

    Glover: Generalizable open-vocabulary affor- dance reasoning for task-oriented grasping,

    T. Maet al., “Glover: Generalizable open-vocabulary affor- dance reasoning for task-oriented grasping,”arXiv preprint arXiv:2411.12286, 2024

    2024 arXiv

  61. [69]

    Interactive planning using large language models for partially observable robotic tasks,

    L. Sunet al., “Interactive planning using large language models for partially observable robotic tasks,” inICRA, pp. 14054–14061, 2024

    2024

  62. [70]

    3d-llm: Injecting the 3d world into large language models,

    Y . Honget al., “3d-llm: Injecting the 3d world into large language models,” inNeurIPS, vol. 36, pp. 20482–20494, 2023

    2023

  63. [71]

    Is your lidar placement optimized for 3d scene understanding?,

    Y . Liet al., “Is your lidar placement optimized for 3d scene understanding?,” inNeurIPS, vol. 37, pp. 34980–35017, 2024

    2024

  64. [72]

    G3-lq: Marrying hyperbolic alignment with explicit semantic-geometric modeling for 3d visual ground- ing,

    Y . Wanget al., “G3-lq: Marrying hyperbolic alignment with explicit semantic-geometric modeling for 3d visual ground- ing,” inCVPR, pp. 13917–13926, 2024

    2024

  65. [73]

    Learning transferable visual models from natural language supervision,

    A. Radfordet al., “Learning transferable visual models from natural language supervision,” inICML, pp. 8748–8763, 2021

    2021

  66. [74]

    Vlm-grounder: A vlm agent for zero-shot 3d visual grounding,

    R. Xuet al., “Vlm-grounder: A vlm agent for zero-shot 3d visual grounding,”arXiv preprint arXiv:2410.13860, 2024

    2024 arXiv

  67. [75]

    Text-guided graph neural networks for referring 3d instance segmentation,

    P.-H. Huanget al., “Text-guided graph neural networks for referring 3d instance segmentation,” inAAAI, vol. 35, pp. 1610–1618, 2021

    2021

  68. [76]

    Mikasa: Multi-key-anchor & scene- aware transformer for 3d visual grounding,

    C.-P. Changet al., “Mikasa: Multi-key-anchor & scene- aware transformer for 3d visual grounding,” inCVPR, pp. 14131–14140, 2024

    2024

  69. [77]

    Language conditioned spatial relation rea- soning for 3d object grounding,

    S. Chenet al., “Language conditioned spatial relation rea- soning for 3d object grounding,” inNeurIPS, 2022

    2022

  70. [78]

    Mask3d: Mask transformer for 3d semantic instance segmentation,

    J. Schultet al., “Mask3d: Mask transformer for 3d semantic instance segmentation,” inICRA, pp. 8216–8223, 2023. 9

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.