Pith. sign in

REVIEW 3 major objections 5 minor 92 references

RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By supervising positions of all nouns in a referring expression, RG-SAN raises 3D segmentation mIoU by 5.1 points.

desk verdict The 5.1-point mIoU gain looks real and well-ablated, but the 'solely spatial' claim and the unvalidated dependency-rule target selection are the parts a referee should drill into. read the letter →

arxiv 2412.02402 v2 pith:6UADAWSQ submitted 2024-12-03 cs.CV

Changli Wu , Qi Chen , Jiayi Ji , Haowei Wang , Yiwei Ma , You Huang , Gen Luo , Hao Fei
show 2 more authors
Xiaoshuai Sun Rongrong Ji
This is my paper · ORCID
classification cs.CV
keywords 3Dreferringexpressionsegmentation3D-RESpointcloudspatialreasoningweaklysupervisedlearningdependencyparsingcomprehensionScanRefer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that 3D referring expression segmentation fails mainly because models match text to point clouds without knowing where the things named in the text are, and that this can be fixed by supervising spatial positions of all mentioned nouns using only the target object's location as ground truth. RG-SAN does this with a Text-driven Localization Module that gives every noun token a 3D coordinate and refines it across rounds of multimodal attention, and a Rule-guided Weak Supervision strategy that uses dependency-tree rules to pick the target noun and trains its position with an L1 loss while mask losses train its segmentation kernel. On the ScanRefer benchmark the paper reports an overall mIoU of 44.6, an increase of 5.1 points over the previous single-task state of the art, with the largest gain in the 'Multiple' setting where same-class distractors are present. If correct, the paper establishes that spatial position, not just semantic alignment, is the missing ingredient for referring segmentation in 3D scenes.

What carries the argument

The load-bearing mechanism is a pair of modules. The Text-driven Localization Module (TLM) computes a similarity distribution between each word token and every superpoint, initializes each token's 3D position as the attention-weighted average of superpoint positions, and then iteratively refines those positions through MLP-predicted offsets, injecting absolute positional encodings into self-attention and table-based relative positional encodings into cross-attention. The Rule-guided Weak Supervision (RWS) strategy uses hand-written dependency-tree rules (Algorithm 1) to select the target noun among the nouns mentioned in the sentence, so that only that token's position receives the L1 position loss against the target's ground-truth center, while BCE and Dice losses train the response map formed by multiplying the target word's segment kernel with superpoint features. Together they let the network learn spatial relations among all entities from a single supervision signal: the target's location.

What would settle it

Run the ScanRefer validation set with the rule-selected target noun replaced by the human-annotated target noun: if the mIoU gap is small, the rules are not the load-bearing component and the stated mechanism is misidentified; if the gap is large, the reported 5.1-point gain depends on the unvalidated rule set and could shrink under a different selector.

Watch

Extended reading notes

Core claim

Stated on the paper's own terms, RG-SAN is the claim that a single end-to-end network can segment a referred 3D object more accurately by first localizing every entity mentioned in the expression and then letting the target's known position pull the rest of the reasoning into place. The network's mask prediction uses the segment kernel of the target word, obtained after iterative text–point-cloud interaction with absolute and relative positional encodings; the position of that same kernel is supervised by the ground-truth center of the target instance, and no position label is used for the other nouns. The experiments report that this scheme outperforms the previous single-task method 3D-STMN by 5.1 mIoU overall (44.6 versus 39.5) and by 6.3 mIoU in the Multiple setting (37.4 versus 31.1), while adding only about 12 ms of inference time, and that the same modules transfer to ReferIt3D, where Sr3D mIoU rises by 5.2 points and Nr3D by 1.0 point. The paper also shows qualitatively that the model segments auxiliary nouns such as a 'coat' or 'couches' even when those categories are not in the training labels.

Load-bearing premise

The whole Rule-guided Weak Supervision scheme assumes the hand-written dependency-tree rules identify the correct target noun in nearly every expression, because a wrong choice makes the position loss and the segmentation mask supervise the wrong entity.

Editorial extensions

If this is right

  • On ScanRefer, RG-SAN's reported 44.6 overall mIoU is 5.1 points above the prior single-task SOTA, with Acc@0.25 up 7.1 points, at an inference cost only 12 ms higher.
  • The largest improvement is in the Multiple setting (37.4 vs 31.1 mIoU, +6.3), where the target is surrounded by same-class distractors—the regime that most needs spatial disambiguation.
  • Ablations attribute the gains to both modules: removing TLM drops overall mIoU from 42.3 to 40.3, and replacing RWS's rule-guided target selection with attention-based Top1 drops it from 44.6 to 42.3.
  • The same rules and losses transfer to ReferIt3D (Sr3D +5.2 mIoU, Nr3D +1.0 mIoU), suggesting the benefit is not specific to ScanRefer.
  • Qualitative results show the model localizes and segments non-target nouns (e.g., 'coat', 'couches'), indicating fine-grained word–superpoint alignment beyond the target object.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because about 92% of ScanRefer expressions contain spatial terms (the paper's own appendix estimate), the reported gains likely reflect a benchmark that is heavily spatial; on a non-spatial referring dataset the gap over 3D-STMN could shrink.
  • The rule-based target selector is the part least validated: Appendix E reports only about 80% agreement with an LLM on target-word identification, so the rules' failure cases are not quantified against human annotations; an oracle-target ablation would show how much headroom remains.
  • If position supervision is the active ingredient, then extending it from a single point to multiple points per noun could address the plural-noun failures the paper lists as a limitation and may further improve 'between/among' expressions.
  • The ability to segment nouns that never appear in the training labels suggests the same weakly supervised alignment could serve open-vocabulary 3D instance segmentation, an application the paper mentions only in passing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes RG-SAN, an end-to-end method for 3D referring expression segmentation on point clouds. The method consists of a Text-driven Localization Module (TLM) that predicts and iteratively refines 3D positions for all noun entities mentioned in the referring expression, and a Rule-guided Weak Supervision (RWS) strategy that uses dependency-tree rules (Algorithm 1) to select the target noun token, whose predicted position is supervised with an L1 loss against the ground-truth object center while the corresponding text kernel produces the segmentation mask. Experiments on ScanRefer report an overall mIoU of 44.6 versus 39.5 for the prior single-task state of the art 3D-STMN, with larger gains in the "Multiple" setting; additional results are reported on ReferIt3D. The central claim is that explicitly modeling spatial positions of all mentioned entities is what drives the improvement.

Significance. If the reported results hold, RG-SAN would be the strongest single-task method on ScanRefer for 3D-RES, and the design principle of jointly localizing text entities in 3D space is a sensible and potentially reusable idea. The paper ships code, evaluates on an external benchmark, and includes a thorough set of ablations (Tables 2-5 and appendix) that consistently isolate the contributions of the proposed modules. The main risk is that the attribution of the gains to correct spatial localization rests on the hand-written rule-based target selection, which is not directly validated and has known failure modes; if those failures are frequent, the mechanism story is not established, even if the empirical leaderboard numbers happen to be reproducible.

major comments (3)
  1. [Sec. 3.3.1 (Algorithm 1) and Appendix E] The accuracy of Rule-guided Target Selection is load-bearing but not directly validated. Algorithm 1 identifies the target token whose position is supervised by Eq. (16) and whose kernel produces the mask in Eq. (11), and Table 4 shows that RTS contributes about 2.1 mIoU over the Root baseline. However, the rule set is structurally fragile: line 3 moves to any nsubj or compound dependent, so for a sentence like "The desk has a chair next to it" it selects 'desk' rather than 'chair', and for multi-word class names such as 'coffee table' it selects the modifier 'coffee' instead of the head 'table'. Appendix E reports only about 80% agreement with LLAMA 2 70B on target-word identification, and the statement that RWS is more accurate on the remaining 20% is asserted without evidence. Since a nontrivial fraction of wrong target selections would supervise the position of the wrong entity and draw the segmentation kernel from the wrong word, the paper needs a quantitative validation of Algorithm 1 on the ScanRefer expressions (e.g., human-annotated target token agreement on a random sample, or comparison against available instance-class annotations), plus an analysis of the failure cases and their effect on the position loss and final masks.
  2. [Abstract, Sec. 1, and Sec. 3.3.2] The claim that the method uses "solely the spatial information of the target instance for supervision" is contradicted by the training objectives. Equations (13)-(15) apply BCE and Dice losses between the predicted response map and the ground-truth target mask, and Eq. (17) combines these with the position loss. The target instance therefore receives full mask supervision, not solely spatial-position supervision. The wording should be revised to state that spatial position is supervised only for the target, while all entities receive spatial position predictions from the TLM; otherwise the abstract overstates the novelty of the supervision scheme.
  3. [Sec. 4.3 and checklist item 7] The main quantitative claim of a 5.1-point mIoU improvement over 3D-STMN is based on a single run with no error bars, as acknowledged in the checklist. Given that the differences in some metrics are small (e.g., Unique mIoU is identical at 74.5), the paper should report at least three seeds with standard deviations for the main Table 1 results, or otherwise justify that the comparison is not within run-to-run noise. This is particularly important for the 'Multiple' setting where the largest claimed gain (6.3 mIoU) occurs.
minor comments (5)
  1. [Sec. 3.3.2 (Eq. 17)] The sentence defining the hyperparameters lists "λbce, λdice, λrel and λscore," but Eq. (17) contains λpos and no λrel; this looks like a typo and should be corrected.
  2. [Algorithm 1] The notation "Ei" in lines 2, 7, and 11 is not defined; the algorithm should specify that it is iterating over edges or relations with head Vi, and the index variable should be made explicit.
  3. [Appendix C.3 (Table 9)] The column header in Table 9 reads "Text Encoder" but the table compares visual backbones (SSTNet, PointNet++, SPFormer); the header should read "Visual Encoder" or "Backbone."
  4. [Fig. 6(f) and Appendix F] The qualitative claim that RG-SAN "is even able to accurately recognize the plural form of the entity noun 'couches'" appears to be in tension with Appendix F, which states the method has difficulty localizing plural nouns because it predicts a single point per token; the distinction between recognizing an auxiliary plural and localizing a plural target should be clarified.
  5. [Sec. 4.4.3 (Table 4)] The text says the Root strategy "slightly outperforms" the w/o RWS baseline, and the table indeed shows a 0.2 mIoU gap in Overall; the interpretation is fine, but the wording could be more precise about which metric is being compared.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the SOTA claim is benchmarked against external ScanRefer ground truth; the only self-citations (DDI, 3D-STMN) are disclosed and not load-bearing.

full rationale

RG-SAN's central claim is an empirical benchmark result: Table 1 reports 44.6 vs 39.5 mIoU against 3D-STMN on ScanRefer, with ground-truth target masks used for BCE/Dice (Eqs. 13-15) and ground-truth target position for Lpos (Eq. 16). The predicted mask is formed as sigma(E_tgt * S^T) (Eq. 11); no quantity in this chain is defined in terms of the method's own output or fitted to the evaluation metric. The rule-guided target selection (Algorithm 1) is a hand-written heuristic that determines which kernel is supervised; its accuracy is a correctness/robustness concern, not a circularity, because the final evaluation does not depend on the rules' self-consistency. Appendix E's claim that RWS is more accurate than LLAMA-2 on the ~20% disagreement is asserted without quantitative evidence—this is a missing-support flag, not a circularity. The only self-citations are [65] for the DDI building block and the 'w/o RWS' attention-Top1 baseline; both are disclosed components and ablations, and the headline gain does not reduce to them. lambda_pos is tuned by ablation (Table 5), which is standard hyperparameter selection, not a fitted input advertised as a prediction. Overall score 2 reflects one minor non-load-bearing self-citation; no derivation-from-input circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on tuned hyperparameters (position-loss weight, number of rounds, binarization threshold) and on assumptions about the dataset and the dependency rules, rather than on a derived theory. No new physical or conceptual entities are introduced.

free parameters (3)
  • Position loss weight λpos = 0.5
    Tuned by ablation in Table 5; overall mIoU peaks at weight 0.5.
  • Number of TLM rounds L = 6
    Tuned by ablation in Appendix C.1; overall mIoU peaks at six rounds.
  • Mask binarization threshold = 0.5
    Used in Eqs. 12 and 13 for prediction and ground-truth binarization; a standard threshold, not tuned.
assumptions (4)
  • domain assumption The ScanRefer dataset is heavily spatial (about 92% of descriptions contain spatial relations), so spatial modeling is central.
    Appendix A motivates the method; if the dataset were mostly non-spatial, the expected gains would shrink.
  • ad hoc to paper The hand-written dependency-tree rules in Algorithm 1 identify the target noun for most expressions.
    The rules are designed by the authors and are not quantitatively validated on the full dataset; Appendix E reports only about 80% agreement with an LLM, implying a non-trivial failure rate.
  • domain assumption Superpoint centroids (Pgt) are reliable spatial supervision for the target object.
    Eq. 16 uses the center of the target's superpoints as regression target; assumes this captures the linguistically relevant position.
  • ad hoc to paper Iterative offset refinement converges to meaningful positions within six rounds.
    Eq. 5 and Appendix C.1 show performance peaks at six rounds; the assumption is that the MLP offset updates remain stable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation." pith.science (2026). https://pith.science/paper/6UADAWSQ

@misc{pith2026241202402,
  author       = {Pith},
  title        = {Pith review of: RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6UADAWSQ}},
  note         = {Machine review of arXiv:2412.02402}
}
read the original abstract

3D Referring Expression Segmentation (3D-RES) aims to segment 3D objects by correlating referring expressions with point clouds. However, traditional approaches frequently encounter issues like over-segmentation or mis-segmentation, due to insufficient emphasis on spatial information of instances. In this paper, we introduce a Rule-Guided Spatial Awareness Network (RG-SAN) by utilizing solely the spatial information of the target instance for supervision. This approach enables the network to accurately depict the spatial relationships among all entities described in the text, thus enhancing the reasoning capabilities. The RG-SAN consists of the Text-driven Localization Module (TLM) and the Rule-guided Weak Supervision (RWS) strategy. The TLM initially locates all mentioned instances and iteratively refines their positional information. The RWS strategy, acknowledging that only target objects have supervised positional information, employs dependency tree rules to precisely guide the core instance's positioning. Extensive testing on the ScanRefer benchmark has shown that RG-SAN not only establishes new performance benchmarks, with an mIoU increase of 5.1 points, but also exhibits significant improvements in robustness when processing descriptions with spatial ambiguity. All codes are available at https://github.com/sosppxo/RG-SAN.

Figures

Figures reproduced from arXiv: 2412.02402 by the authors.

Figure 1
Figure 1. Illustration with a target object and mul [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview of the proposed RG-SAN. This model analyzes a point cloud and a textual [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of all the nouns in the textual description. Our RG-SAN can segment instances [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Statistics of samples in the ScanRefer dataset based on the presence of spatial relation descriptions, where “spatial” represents samples with spa￾tially related descriptions, while “w/o spatial” denotes spatially unrelated sam￾ples. We extended the 3D-RES task on the …
Figure 5
Figure 5. Figure 5: Qualitative comparison between the proposed RG-SAN and 3D-STMN. [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: The visualization of the predicted masks of mentioned instances of our RG-SAN. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 55 canonical work pages

  1. [1]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16 , pages 422–440. Springer, 2020

    2020

  2. [2]

    Localizing moments in video with natural language

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803–5812, 2017

    2017

  3. [3]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129–19139, 2022

    2022

  4. [4]

    WeaQA: Weak Supervision via Captions for Visual Question Answering

    Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral. Weaqa: Weak supervision via captions for visual question answering. arXiv preprint arXiv:2012.02356, 2020

    work page Pith review arXiv 2012

  5. [5]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202–221. Springer, 2020

    2020

  6. [6]

    Language conditioned spatial relation reasoning for 3d object grounding

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in Neural Information Processing Systems, 35:20522–20535, 2022

    2022

  7. [7]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017

    2017

  8. [8]

    Bert: Pre-training of deep bidirec- tional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

Show all 92 references
  1. [9]

    Weak supervision and referring attention for temporal-textual association learning

    Zhiyuan Fang, Shu Kong, Zhe Wang, Charless Fowlkes, and Yezhou Yang. Weak supervision and referring attention for temporal-textual association learning. arXiv preprint arXiv:2006.11747, 2020

    2006 arXiv

  2. [10]

    Video-of-thought: Step-by-step video reasoning from perception to cognition

    Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. In Forty-first International Conference on Machine Learning, 2024

    2024

  3. [11]

    Vitron: A unified pixel-level vision llm for understanding, generating, segmenting, editing, 2024

    Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision llm for understanding, generating, segmenting, editing, 2024

    2024

  4. [12]

    Enhancing video-language representations with structural spatio-temporal alignment

    Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. Enhancing video-language representations with structural spatio-temporal alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  5. [13]

    Free-form description guided 3d visual graph network for object grounding in point cloud

    Mingtao Feng, Zhen Li, Qi Li, Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang, and Ajmal Mian. Free-form description guided 3d visual graph network for object grounding in point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Visi...

    2021

  6. [14]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024

    2024 arXiv

  7. [15]

    Structured multi- modal feature embedding and alignment for image-sentence retrieval

    Xuri Ge, Fuhai Chen, Joemon M Jose, Zhilong Ji, Zhongqin Wu, and Xiao Liu. Structured multi- modal feature embedding and alignment for image-sentence retrieval. In Proceedings of the 29th ACM international conference on multimedia, pages 5185–5193, 2021

    2021

  8. [16]

    3shnet: Boosting image–sentence retrieval via visual semantic–spatial self-highlighting

    Xuri Ge, Songpei Xu, Fuhai Chen, Jie Wang, Guoxin Wang, Shan An, and Joemon M Jose. 3shnet: Boosting image–sentence retrieval via visual semantic–spatial self-highlighting. Information Processing & Management, 61(4):103716, 2024

    2024

  9. [17]

    Vqa-lol: Visual question answering under the lens of logic

    Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Vqa-lol: Visual question answering under the lens of logic. In European conference on computer vision, pages 379–396. Springer, 2020

    2020

  10. [18]

    Person re-identification method based on color attack and joint defence

    Yunpeng Gong, Liqing Huang, and Lifei Chen. Person re-identification method based on color attack and joint defence. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4313–4322, 2022. 12

    2022

  11. [19]

    3d semantic segmentation with submanifold sparse convolutional networks

    Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9224–9232, 2018

    2018

  12. [20]

    Segpoint: Segment any point cloud via large language model

    Shuting He, Henghui Ding, Xudong Jiang, and Bihan Wen. Segpoint: Segment any point cloud via large language model. In European Conference on Computer Vision, pages 349–367. Springer, 2025

    2025

  13. [21]

    3d-llm: Injecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494, 2023

    2023

  14. [22]

    Chat-3d v2: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint arXiv:2312.08168, 2023

    2023 arXiv

  15. [23]

    Reason3d: Searching and reasoning 3d segmentation via large language model

    Kuan-Chih Huang, Xiangtai Li, Lu Qi, Shuicheng Yan, and Ming-Hsuan Yang. Reason3d: Searching and reasoning 3d segmentation via large language model. arXiv preprint arXiv:2405.17427, 2024

    2024 arXiv

  16. [24]

    Text-guided graph neural networks for referring 3d instance segmentation

    Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1610–1618, 2021

    2021

  17. [25]

    Referring image segmentation via joint mask contextual embedding learning and progressive alignment network

    Ziling Huang and Shin’ichi Satoh. Referring image segmentation via joint mask contextual embedding learning and progressive alignment network. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro...

    2023 doi

  18. [26]

    Bottom up top down detection transformers for language grounding in images and point clouds

    Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In European Conference on Computer Vision, pages 417–433. Springer, 2022

    2022

  19. [27]

    Comprehensive multi-modal interactions for referring image segmentation

    Kanishk Jain and Vineet Gandhi. Comprehensive multi-modal interactions for referring image segmentation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3427–3435, 2022

    2022

  20. [28]

    Weak supervision helps emergence of word-object alignment and improves vision-language tasks.arXiv preprint arXiv:1912.03063, 2019

    Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. Weak supervision helps emergence of word-object alignment and improves vision-language tasks.arXiv preprint arXiv:1912.03063, 2019

    1912 arXiv

  21. [29]

    Flexible visual grounding

    Yongmin Kim, Chenhui Chu, and Sadao Kurohashi. Flexible visual grounding. In Samuel Louvan, Andrea Madotto, and Brielen Madureira, editors, Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics: Student Research Workshop , pages 285–299, Dub...

    2022 doi

  22. [30]

    Stratified transformer for 3d point cloud segmentation

    Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8500–8509, 2022

    2022

  23. [31]

    Mask-attention-free transformer for 3d instance segmentation

    Xin Lai, Yuhui Yuan, Ruihang Chu, Yukang Chen, Han Hu, and Jiaya Jia. Mask-attention-free transformer for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3693–3703, 2023

    2023

  24. [32]

    Large-scale point cloud semantic segmentation with superpoint graphs

    Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4558–4567, 2018

    2018

  25. [33]

    Weakly supervised referring image segmentation with intra-chunk and inter-chunk consistency

    Jungbeom Lee, Sungjin Lee, Jinseok Nam, Seunghak Yu, Jaeyoung Do, and Tara Taghavi. Weakly supervised referring image segmentation with intra-chunk and inter-chunk consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21870–21881, 2023

    2023

  26. [34]

    Fully and weakly supervised referring expression segmentation with end-to-end learning

    Hui Li, Mingjie Sun, Jimin Xiao, Eng Gee Lim, and Yao Zhao. Fully and weakly supervised referring expression segmentation with end-to-end learning. IEEE Transactions on Circuits and Systems for Video Technology, 2023

    2023

  27. [35]

    Fine-grained semantically aligned vision-language pre-training

    Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang. Fine-grained semantically aligned vision-language pre-training. Advances in neural information processing systems, 35:7290–7303, 2022. 13

    2022

  28. [36]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  29. [37]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  30. [38]

    Toist: Task oriented instance segmentation transformer with noun-pronoun distillation

    Pengfei Li, Beiwen Tian, Yongliang Shi, Xiaoxue Chen, Hao Zhao, Guyue Zhou, and Ya-Qin Zhang. Toist: Task oriented instance segmentation transformer with noun-pronoun distillation. Advances in Neural Information Processing Systems, 35:17597–17611, 2022

    2022

  31. [39]

    Understanding embodied reference with touch-line transformer

    Yang Li, Xiaoxue Chen, Hao Zhao, Jiangtao Gong, Guyue Zhou, Federico Rossano, and Yixin Zhu. Understanding embodied reference with touch-line transformer. In ICLR, 2023

    2023

  32. [40]

    Transformer-empowered invariant grounding for video question answering

    Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Transformer-empowered invariant grounding for video question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

    2023

  33. [41]

    Laso: Language-guided affordance segmentation on 3d object

    Yicong Li, Na Zhao, Junbin Xiao, Chun Feng, Xiang Wang, and Tat-seng Chua. Laso: Language-guided affordance segmentation on 3d object. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14251–14260, 2024

    2024

  34. [42]

    Instance segmentation in 3d scenes using semantic superpoint tree networks

    Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, and Kui Jia. Instance segmentation in 3d scenes using semantic superpoint tree networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2783–2792, 2021

    2021

  35. [43]

    A unified framework for 3d point cloud visual grounding

    Haojia Lin, Yongdong Luo, Xiawu Zheng, Lijiang Li, Fei Chao, Taisong Jin, Donghao Luo, Chengjie Wang, Yan Wang, and Liujuan Cao. A unified framework for 3d point cloud visual grounding. arXiv preprint arXiv:2308.11887, 2023

    2023 arXiv

  36. [44]

    Referring image segmentation using text supervision

    Fang Liu, Yuhao Liu, Yuqiu Kong, Ke Xu, Lihe Zhang, Baocai Yin, Gerhard Hancke, and Rynson Lau. Referring image segmentation using text supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22124–22134, 2023

    2023

  37. [45]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  38. [46]

    Scaneru: Interactive 3d visual grounding based on embodied reference understanding

    Ziyang Lu, Yunqiang Pei, Guoqing Wang, Peiwei Li, Yang Yang, Yinjie Lei, and Heng Tao Shen. Scaneru: Interactive 3d visual grounding based on embodied reference understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3936–3944, 2024

    2024

  39. [47]

    3d-sps: Single-stage 3d visual grounding via referred point progressive selection

    Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16454–16...

    2022

  40. [48]

    The stanford corenlp natural language processing toolkit

    Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pa...

    2014

  41. [49]

    V-net: Fully convolutional neural networks for volumetric medical image segmentation

    Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565–571. Ieee, 2016

    2016

  42. [50]

    Weakly supervised video moment retrieval from text queries

    Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy-Chowdhury. Weakly supervised video moment retrieval from text queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11592–11601, 2019

    2019

  43. [51]

    Bridging the gap between 2d and 3d visual question answering: A fusion approach for 3d vqa

    Wentao Mo and Yang Liu. Bridging the gap between 2d and 3d visual question answering: A fusion approach for 3d vqa. arXiv preprint arXiv:2402.15933, 2024

    2024 arXiv

  44. [52]

    Deep hough voting for 3d object detection in point clouds

    Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277–9286, 2019

    2019

  45. [53]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017. 14

    2017

  46. [54]

    X-refseg3d: Enhancing referring 3d instance segmentation via structured cross-modal graph neural networks

    Zhipeng Qian, Yiwei Ma, Jiayi Ji, and Xiaoshuai Sun. X-refseg3d: Enhancing referring 3d instance segmentation via structured cross-modal graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4551–4559, 2024

    2024

  47. [55]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  48. [56]

    You only look once: Unified, real-time object detection

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016

    2016

  49. [57]

    Mask3d for 3d semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d for 3d semantic instance segmentation. arXiv preprint arXiv:2210.03105, 2022

    2022 arXiv

  50. [58]

    Mpnet: Masked and permuted pre-training for language understanding

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems, 33:16857–16867, 2020

    2020

  51. [59]

    ReCLIP: A strong zero-shot baseline for referring expression comprehension

    Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. ReCLIP: A strong zero-shot baseline for referring expression comprehension. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual ...

    2022 doi

  52. [60]

    Superpoint transformer for 3d scene instance segmentation

    Jiahao Sun, Chunmei Qing, Junpeng Tan, and Xiangmin Xu. Superpoint transformer for 3d scene instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 2393–2401, 2023

    2023

  53. [61]

    Text augmented spatial aware zero-shot referring image segmentation

    Yucheng Suo, Linchao Zhu, and Yi Yang. Text augmented spatial aware zero-shot referring image segmentation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1032–1043, Singapore, December 2023. ...

    2023 doi

  54. [62]

    Interpretable counting for visual question answering

    Alexander Trott, Caiming Xiong, and Richard Socher. Interpretable counting for visual question answering. arXiv preprint arXiv:1712.08697, 2017

    2017 arXiv

  55. [63]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  56. [64]

    Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes

    Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08769, 2023

    2023 arXiv

  57. [65]

    3d-stmn: Dependency-driven superpoint-text matching network for end-to-end 3d referring expression segmentation

    Changli Wu, Yiwei Ma, Qi Chen, Haowei Wang, Gen Luo, Jiayi Ji, and Xiaoshuai Sun. 3d-stmn: Dependency-driven superpoint-text matching network for end-to-end 3d referring expression segmentation. arXiv preprint arXiv:2308.16632, 2023

    2023 arXiv

  58. [66]

    3d-gres: Generalized 3d referring expression segmentation, 2024

    Changli Wu, Yihang Liu, Jiayi Ji, Yiwei Ma, Haowei Wang, Gen Luo, Henghui Ding, Xiaoshuai Sun, and Rongrong Ji. 3d-gres: Generalized 3d referring expression segmentation, 2024. URL https: //arxiv.org/abs/2407.20664

    2024 arXiv

  59. [67]

    Rethinking and improving relative position encoding for vision transformer

    Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10033–10041, 2021

    2021

  60. [68]

    Towards semantic equivalence of tokenization in multimodal llm

    Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards semantic equivalence of tokenization in multimodal llm. arXiv preprint arXiv:2406.05127, 2024

    2024 arXiv

  61. [69]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. In Proceedings of the International Conference on Machine Learning, 2024

    2024

  62. [70]

    Eda: Explicit text-decoupling and dense alignment for 3d visual grounding

    Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. Eda: Explicit text-decoupling and dense alignment for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19231–19242, 2023. 15

    2023

  63. [71]

    A unified framework for 3d scene understanding

    Wei Xu, Chunsheng Shi, Sifan Tu, Xin Zhou, Dingkang Liang, and Xiang Bai. A unified framework for 3d scene understanding. arXiv preprint arXiv:2407.03263, 2024

    2024 arXiv

  64. [72]

    Sat: 2d semantics assisted training for 3d visual grounding

    Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: 2d semantics assisted training for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1856–1866, 2021

    2021

  65. [73]

    In- stancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring

    Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. In- stancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Con...

    2021

  66. [74]

    CK-transformer: Commonsense knowledge enhanced transformers for referring expression comprehension

    Zhi Zhang, Helen Yannakoudakis, Xiantong Zhen, and Ekaterina Shutova. CK-transformer: Commonsense knowledge enhanced transformers for referring expression comprehension. In Andreas Vlachos and Isabelle Augenstein, editors, Findings of the Association for Computational Linguist...

    2023 doi

  67. [75]

    3dvg-transformer: Relation modeling for visual grounding on point clouds

    Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928–2937, 2021

    2021

  68. [76]

    Towards learning a generalist model for embodied navigation

    Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang. Towards learning a generalist model for embodied navigation. arXiv preprint arXiv:2312.02010, 2023

    2023 arXiv

  69. [77]

    left”, “right

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911–2921, 2023. 16 Appendix A The Critical Role of...

    2023

  70. [78]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and introduction accurately summarize the paper’s key contribu- tions and scope. Guidelines: • The answer...

  71. [79]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of the work in the Appendix. Guidelines: • The answer NA means that the paper has no limitation while the answer No means ...

  72. [80]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 22 Justification: The paper primarily focuses on the experimental exploration of model struc- tures rather t...

  73. [81]

    Detailed experimental settings are provided in Sec

    Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  74. [82]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The paper provides open acc...

  75. [83]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: The paper specifies all the n...

  76. [84]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: The paper presents the results from a single run fo...

  77. [85]

    4 and Appendix

    Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: The paper provides deta...

  78. [86]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The research conducted in the paper adheres to the NeurIPS Code of Ethics, ensu...

  79. [87]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] 25 Justification: The paper includes a discussion on the potential societal impacts of the work in the Appendix. Guideli...

  80. [88]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The research ex...

  81. [89]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The paper credit...

  82. [90]

    Guidelines: • The answer NA means that the paper does not release new assets

    New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not introduce any new assets. Guidelines: • The answer NA means that the paper does not release new as...

  83. [91]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  84. [92]

    Guidelines: 27 • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.