Pith. sign in

REVIEW 3 major objections 7 minor 55 references

Voxel backbone and scene-mixing boost 3D captioning by 11-14%

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-07-08 16:28 UTC pith:L2YG7RYG

load-bearing objection Solid 3D dense captioning paper with large gains, but the main ablation confounds architecture with backbone/detector upgrades the 3 major comments →

arxiv 2607.06097 v1 pith:L2YG7RYG submitted 2026-07-07 cs.CV cs.AI

PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNet

Xiaopei Wu , Chenshu Hou , Liang Peng , Dan Xu , Binbin Lin , Xiaoshui Huang , Yuenan Hou , Yu Li
show 4 more authors
Wenxiao Wang Haifeng Liu Deng Cai Wanli Ouyang
This is my paper
classification cs.CV cs.AI
keywords 3D dense captioningvoxel-based architecturedata augmentationpseudo-labelingteacher-student frameworkpoint cloud understandinginstance mixingscene generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that 3D dense captioning — the task of generating natural language descriptions for every object in a 3D point cloud scene — has been held back by two problems: weak data augmentation that never changes spatial layouts, and weak network backbones that feed impoverished features to the caption generator. The authors propose PVCap, which attacks both problems simultaneously. The first component, PseudoCap, builds an instance database by cropping individual objects from all training scenes, then recombines them into synthetic 'pseudo frames' with novel spatial arrangements. Because rearranging objects invalidates their original captions, a frozen teacher model generates pseudo-caption labels for these synthetic scenes, which are filtered for quality and used to train the student model. The second component, VoxelCapNet, is the first voxel-based architecture for 3D dense captioning: it replaces the point-based backbone and detection head used in prior work with a voxel-based backbone (Swin3D) and detection head (CAGroup3D), then adapts the caption head to consume voxel features directly. Together, the two components achieve 89.57% CIDEr@0.5IoU on ScanRefer and 61.61% on Nr3D, surpassing the previous state-of-the-art by 11.41% and 13.99% respectively. The ablation attributes the larger share of the gain (+7.78%) to VoxelCapNet and a smaller but still meaningful share (+2.31%) to PseudoCap.

Core claim

The central claim is that two changes — (1) generating synthetic training scenes by mixing instances across different real scenes to create diverse spatial layouts, and (2) replacing the point-based backbone and detector with a voxel-based architecture that supplies richer features to the caption head — together produce a large improvement in 3D dense captioning quality. The voxel-based architecture contributes the majority of the gain, while the scene-mixing augmentation adds a further increment. The paper also shows that simply training longer without PseudoCap degrades performance, suggesting the improvement from PseudoCap is not merely an extended-training artifact.

What carries the argument

PseudoCap: instance database construction → random instance mixing with collision detection → teacher-student pseudo-label generation. VoxelCapNet: voxelization → voxel backbone (Swin3D or SparseUNet) → voxel-based detection head (CAGroup3D) with confidence filtering and NMS → caption head adapted to consume filtered voxel features as queries and backbone voxel features as context.

Load-bearing premise

The ablation comparing the baseline (PointNet++ backbone + 3DETR detector) against VoxelCapNet (Swin3D backbone + CAGroup3D detector) changes the backbone, the detector, and the caption-head adaptation simultaneously. The paper does not isolate whether the voxel-based caption-head design or simply the stronger backbone and detector is responsible for the +7.78% gain. If the backbone upgrade accounts for most of the improvement, the conceptual contribution of the voxel-based *

What would settle it

Run the existing Vote2Cap-DETR caption head on top of the Swin3D backbone and CAGroup3D detector (without the voxel-specific caption-head adaptation). If that configuration achieves performance close to VoxelCapNet, the architectural novelty of the caption-head adaptation is not the source of the gain.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If voxel-based architectures are indeed the stronger choice for 3D dense captioning, future work on caption-specific modules (relation reasoning, spatial transformers) should build on voxel features rather than point-based features as the default substrate.
  • The PseudoCap instance-mixing strategy could extend to other 3D vision-language tasks like 3D visual grounding or 3D question answering, where spatial layout diversity also matters.
  • The teacher-student pseudo-labeling approach for handling caption invalidation after scene rearrangement is a general technique — any data augmentation that changes spatial relationships could use the same pipeline to regenerate labels.
  • The finding that backbone capacity directly correlates with caption quality (Table 8) suggests that 3D dense captioning performance may be bottlenecked more by feature quality than by caption-head design, shifting where future architectural innovation should focus.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes PVCap, a framework for 3D dense captioning comprising two contributions: (1) PseudoCap, a data augmentation strategy that mixes instances from different scenes to generate pseudo frames with diverse spatial layouts, supervised via a teacher-student framework; and (2) VoxelCapNet, a voxel-based 3D dense captioning network that adapts the caption head from Vote2Cap-DETR to use voxel features from a modern voxel-based backbone (Swin3D) and detection head (CAGroup3D). The method is evaluated on ScanRefer and Nr3D, reporting substantial improvements over prior state-of-the-art (e.g., +11.41% C@0.5IoU on ScanRefer, +13.99% on Nr3D). Ablation studies cover backbone choice, PseudoCap probability, training epochs, loss weights, NMS threshold, and backbone capacity.

Significance. The paper addresses two practical limitations in 3D dense captioning: limited data augmentation diversity and reliance on weaker backbone/detection components. The PseudoCap augmentation is a reasonable approach to generating diverse spatial layouts, and the teacher-student design for pseudo-label generation is well-motivated. The reported gains on ScanRefer and Nr3D are large. The ablation on PseudoCap probability (Table 6, prob=0 vs prob=0.5) properly controls for extra training epochs, which strengthens the claim that the augmentation itself (not merely longer training) drives the improvement. The code release commitment and the breadth of ablations are commendable. However, the significance of the VoxelCapNet contribution is partially undermined by a confounded ablation (see major comments), which makes it difficult to attribute the gains to the voxel-based caption-head adaptation versus the backbone/detector upgrade.

major comments (3)
  1. §4.3.1, Table 4 (experiments 1→2): The ablation attributing +7.78% C@0.5IoU to VoxelCapNet simultaneously swaps the backbone (PointNet++/3DETR encoder → Swin3D), the detection head (3DETR → CAGroup3D), AND the caption-head adaptation. This confounds three changes, making it impossible to isolate the contribution of the voxel-based caption-head adaptation (the conceptual novelty of VoxelCapNet) from the contribution of simply using a stronger backbone and detector. Table 5 shows that backbone choice alone accounts for a 7.65% spread (PTV2: 61.94% vs Swin3D: 69.59%) within VoxelCapNet, and Table 3 shows detection mAP@0.5 jumps from 52.13% to 61.81% across this same transition. Since C@0.5IoU requires IoU>0.5, better detection mechanically improves captioning scores. The paper should run Vote2Cap-DETR's caption head on top of Swin3D/CAGroup3D (or at minimum, the Swin3D backbone with the 3DE
  2. TR detection head) to isolate the caption-head adaptation's contribution from the backbone/detector upgrade. Without this control, the claim that VoxelCapNet's architectural adaptation contributes +7.78% (§4.3.1) is unverifiable. This is load-bearing for the paper's central claim about VoxelCapNet's novelty.
  3. §3.2.3, Appendix B: The PseudoCap instance mixing places instances at random locations with arbitrary orientations, and Appendix C.1 acknowledges that 'instances in pseudo frames can cross over the wall, and their orientations are arbitrary.' The teacher generates pseudo-labels for these physically implausible scenes. While the paper argues this serves as regularization, there is no analysis of pseudo-label quality on these implausible configurations. A quantitative assessment of teacher pseudo-label quality (e.g., BLEU/CIDEr of pseudo-labels vs. ground-truth captions for the real-frame counterparts of mixed instances, or human evaluation of a sample) would strengthen the claim that PseudoCap provides 'sufficient supervision about spatial relations' (Appendix C.1). This is secondary to the main ablation concern but affects the PseudoCap contribution's credibility.
minor comments (7)
  1. Table 1: The C@0.5IoU value for PVCap-Swin3D under SCST (89.57%) is highlighted as the headline result, but the corresponding R@0.5IoU (59.97%) is slightly lower than PVCap-SparseUNet's R@0.5IoU (60.11% with 2D features). This should be noted or discussed.
  2. §4.1.3: The implementation states 'we first pre-train a VoxelCapNet for 300 epochs' then 'train the student with PseudoCap for another 300 epochs.' It would help to clarify whether the 300-epoch PseudoCap training (Table 4, exp 2→3) uses the same learning rate schedule as the initial 300-epoch pre-training, or a new schedule.
  3. Table 5: The time column for Swin3D is listed as '390 ms' in the text (§4.3.3) but '391 ms' in the table. Minor inconsistency.
  4. §3.3.3: 'Start Of Seqenece' should be 'Start Of Sequence' (typo).
  5. Table 2: The method is listed as 'χ-Tran2Cap' but the reference [50] is 'X-Trans2Cap'. Consistent naming would help.
  6. The paper would benefit from a brief discussion of the memory/compute overhead of PseudoCap training (teacher inference + student training on pseudo frames), since the teacher-student framework doubles the model footprint during training.
  7. Figure 2: The caption mentions 'The caption head follows the design of [6],' but the figure does not clearly show how V_q (filtered by confidence and NMS) differs from the query features in Vote2Cap-DETR. A more detailed visual distinction would aid understanding.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises two major points: (1) the VoxelCapNet ablation (Table 4, experiments 1→2) confounds backbone, detection head, and caption-head adaptation, making it impossible to isolate the contribution of the voxel-based caption-head adaptation from the backbone/detector upgrade; and (2) the PseudoCap contribution would benefit from quantitative assessment of pseudo-label quality on physically implausible mixed scenes. Both points are well-taken. We agree to run the requested control experiment (Vote2Cap-DETR's caption head on Swin3D/CAGroup3D) and to add quantitative pseudo-label quality analysis. We believe these additions will substantially strengthen the paper.

read point-by-point responses

  1. Referee: §4.3.1, Table 4 (experiments 1→2): The ablation attributing +7.78% C@0.5IoU to VoxelCapNet simultaneously swaps the backbone (PointNet++/3DETR encoder → Swin3D), the detection head (3DETR → CAGroup3D), AND the caption-head adaptation. This confounds three changes, making it impossible to isolate the contribution of the voxel-based caption-head adaptation from the contribution of simply using a stronger backbone and detector. The paper should run Vote2Cap-DETR's caption head on top of Swin3D/CAGroup3D (or at minimum, the Swin3D backbone with the 3DETR detection head) to isolate the caption-head adaptation's contribution from the backbone/detector upgrade.

    Authors: The referee is correct that the ablation in Table 4 (experiments 1→2) confounds three simultaneous changes: backbone, detection head, and caption-head adaptation. We agree that this confound weakens the claim that the voxel-based caption-head adaptation itself contributes +7.78%. We will run the requested control experiment: Vote2Cap-DETR's original caption head on top of the Swin3D backbone and CAGroup3D detection head. This will isolate the contribution of the caption-head adaptation (using filtered voxel features V_q as query features and voxel features V_s as contextual features, per §3.3.3) from the backbone/detector upgrade. We will add this as a new row in Table 4 and revise the text in §4.3.1 to accurately attribute gains across the three factors. We acknowledge that the current phrasing 'VoxelCapNet achieves +7.78%' overstates the isolated contribution of the caption-head adaptation and will revise accordingly. We expect the control to show that the backbone/detector upgrade accounts for a substantial portion of the gain (consistent with the 7.65% backbone spread in Table 5 and the detection mAP jump in Table 3), while the caption-head adaptation provides an additional, smaller but meaningful improvement. We will report whatever the data shows honestly. revision: yes

  2. Referee: §3.2.3, Appendix B: The PseudoCap instance mixing places instances at random locations with arbitrary orientations, and Appendix C.1 acknowledges that 'instances in pseudo frames can cross over the wall, and their orientations are arbitrary.' The teacher generates pseudo-labels for these physically implausible scenes. There is no analysis of pseudo-label quality on these implausible configurations. A quantitative assessment of teacher pseudo-label quality would strengthen the claim that PseudoCap provides 'sufficient supervision about spatial relations' (Appendix C.1).

    Authors: We agree that quantitative assessment of pseudo-label quality on physically implausible mixed scenes would strengthen the PseudoCap contribution's credibility. We will conduct the following analysis: (1) For instances that appear in both pseudo frames and their original real frames, we will compute BLEU/CIDEr of the teacher's pseudo-labels against the ground-truth captions from the real-frame counterparts, measuring how well the teacher describes spatial relations in the new (mixed) context. (2) We will perform a human evaluation on a random sample of 100 pseudo frames, asking annotators to rate pseudo-label fluency and spatial-relation accuracy on a 1–5 scale. We will add these results to Appendix C.1. We note that the teacher model is pre-trained on real frames and thus can generate spatial descriptions conditioned on the actual point cloud layout it observes, even when that layout is physically implausible. The regularization benefit likely comes from exposing the student to diverse spatial configurations rather than from the pseudo-labels being perfectly accurate. The quantitative analysis will allow us to state this claim with appropriate calibration. revision: yes

Circularity Check

0 steps flagged

No circularity found: the paper's claims are empirically validated against external benchmarks, and the teacher-student framework uses a pre-trained model on real data rather than circular self-distillation.

full rationale

The paper proposes PVCap, consisting of PseudoCap (data augmentation) and VoxelCapNet (architecture). The central claims are validated against external benchmarks (ScanRefer, Nr3D) with quantitative comparisons to prior methods. The teacher-student framework in PseudoCap is not circular: the teacher is a frozen model pre-trained on the original caption dataset with ground-truth labels, and it generates pseudo-labels for augmented frames with genuinely novel spatial layouts. The student is trained on both real and pseudo frames. This is standard semi-supervised learning, not a self-definitional loop. The ablation in Table 4 compares Vote2Cap-DETR (baseline) against VoxelCapNet and PseudoCap incrementally. While the ablation confounds backbone/detector changes with the caption-head adaptation (a correctness concern), this is not circularity—the performance gains are measured against external test sets, not defined into the inputs. No step in the derivation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

No new entities are postulated. The method uses existing architectural components (voxel backbones, detection heads, caption heads) and standard training techniques (teacher-student, SCST).

free parameters (6)
  • PseudoCap probability p = 0.5
    Probability of selecting a real frame for PseudoCap augmentation; ablated in Table 6, best at 0.5.
  • Detection loss weight alpha = 1.0
    Balances detection and caption losses; ablated in Table 9, best at 1.0.
  • NMS threshold tau = 0.5
    IoU threshold for non-maximum suppression in detection head; ablated in Table 10, best at 0.5.
  • Instances per category in mixing = 1
    Default number controlling how many instances per category are selected for pseudo frames; stated in Appendix B.
  • Collision detection IoU threshold = not specified
    Pre-defined IoU threshold to prevent pasted instances from overlapping; mentioned but exact value not given.
  • Pseudo-label confidence threshold = not specified
    Threshold for filtering low-confidence pseudo caption labels from the teacher; mentioned in §3.2.3 but exact value not provided.
axioms (4)
  • domain assumption Voxel-based backbones provide richer features for 3D dense captioning than point-based backbones like PointNet++.
    Stated in §1 and §3.3; supported empirically by ablation in Table 4 but confounded with backbone capacity differences.
  • domain assumption Diverse spatial layouts in training data improve captioning of spatial relations.
    Stated in §1 and §3.2; supported by PseudoCap ablation in Table 4 (+2.31% C@0.5IoU).
  • domain assumption Teacher-generated pseudo-labels on augmented frames provide useful supervision signal.
    Underlies the PseudoCap teacher-student framework in §3.2.3; supported by Table 6 (prob=0 vs prob=0.5).
  • standard math Standard captioning metrics (CIDEr, BLEU-4, METEOR, ROUGE) combined with IoU thresholds adequately evaluate 3D dense captioning quality.
    Follows established evaluation protocol from Scan2Cap [10]; used throughout §4.

reviewed 2026-07-08 · how reviews work

0 comments
Cite this review

Pith. "Pith review of PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNet." pith.science (2026). https://pith.science/paper/L2YG7RYG

@misc{pith2026260706097,
  author       = {Pith},
  title        = {Pith review of: PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNet},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L2YG7RYG}},
  note         = {Machine review of arXiv:2607.06097}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

3D dense captioning, an emerging vision-language task, aims to generate descriptive sentences for each object in the 3D scene. Despite the impressive results achieved by previous methods, they suffer from two limitations. First, current research often employs global rigid transformations, such as rotation, to augment scenes without changing their spatial layouts. However, diverse spatial layouts are crucial for training a 3D dense captioning model to describe spatial relations between objects. Second, previous works mainly focus on the design of the caption generation pipeline while utilizing a simple network architecture for other components, i.e., backbone and detection head, which is crucial for extracting rich semantic information for captioning. In this paper, we propose PVCap to alleviate the aforementioned problems. Our PVCap consists of PseudoCap and VoxelCapNet. Specifically, PseudoCap employs a random mixing technique on instances within the dataset, generating numerous pseudo frames with diverse spatial layouts at the instance level. By utilizing a teacher-student framework, PseudoCap obtains pseudo caption labels for these pseudo frames. This data augmentation approach significantly increases the number of training samples and enhances the model's ability to describe the environment effectively. Regarding VoxelCapNet, we introduce a robust caption network that utilizes voxel features and adapts the caption head to the voxel-based network architecture. Our VoxelCapNet can serve as a competitive baseline for future research on 3D dense captioning. Extensive experiments are conducted on two prevalent benchmarks, i.e., ScanRefer and Nr3D. Notably, our method surpasses current state-of-the-art by 11.41% and 13.99% in CIDEr@0.5IoU, respectively. Codes will be made publicly available.

Figures

Figures reproduced from arXiv: 2607.06097 by Binbin Lin, Chenshu Hou, Dan Xu, Deng Cai, Haifeng Liu, Liang Peng, Wanli Ouyang, Wenxiao Wang, Xiaopei Wu, Xiaoshui Huang, Yuenan Hou, Yu Li.

Figure 1
Figure 1. Figure 1: The visualization of pseudo frames and their predicted [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Framework overview of our PVCap. Our PVCap consists of two key components: (1) PseudoCap mixes instances from different scenes to generate pseudo frames with diverse spatial layouts, providing extensive training data to enhance the ability of the model on spatial relation captioning. To prepare caption supervision for pseudo frames, we leverage a frozen teacher to generate pseudo caption labels. Finally, t… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative Comparisons. We compare visualization results with Vote2Cap-DETR[6] and showcase the ground-truth captions. In each scene, the wrong descriptions are marked in red and the corresponding correct descriptions are in green. Besides, we underline the surrounding information to exhibit the ability of our PVCap on environment description. sidering the impressive results achieved by SparseUnet[13], PT… view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of pseudo frames generated by the proposed PseudoCap. [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative Comparison on Nr3D Dataset 13 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 55 canonical work pages · 4 internal anchors

  1. [1]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. InComputer Vision–ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 422–440. Springer, 2020. 5, 6

    work page 2020

  2. [2]

    Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. InProceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2005. 5

    work page 2005

  3. [3]

    3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds

    Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16464–16473, 2022. 1, 2, 6

    work page 2022

  4. [4]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. InEuropean conference on computer vision, pages 202–221. Springer, 2020. 3, 5, 6, 7

    work page 2020

  5. [5]

    D 3 net: A unified speaker-listener architecture for 3d dense captioning and visual grounding

    Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and An- gel X Chang. D 3 net: A unified speaker-listener architecture for 3d dense captioning and visual grounding. InEuropean Conference on Computer Vision, pages 487–505. Springer,

    work page

  6. [6]

    End-to-end 3d dense captioning with vote2cap-detr

    Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen. End-to-end 3d dense captioning with vote2cap-detr. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11124– 11133, 2023. 1, 2, 4, 5, 6, 7, 8, 12

    work page 2023

  7. [7]

    V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning, 2023

    Sijin Chen, Hongyuan Zhu, Mingsheng Li, Xin Chen, Peng Guo, Yinjie Lei, Gang Yu, Taihao Li, and Tao Chen. V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning, 2023. 1, 2, 6, 7

    work page 2023

  8. [8]

    Segment and Select: Vision-Language Segmentation in 3D Scenarios

    Yulin Chen, Zhihang Zhong, and Yuenan Hou. Segment and select: Vision-language segmentation in 3d scenarios.arXiv preprint arXiv:2606.10594, 2026. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  9. [9]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. InEuropean conference on computer vision, pages 104–120. Springer,

    work page

  10. [10]

    Scan2cap: Context-aware dense captioning in rgb- d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb- d scans. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3193–3203,

    work page

  11. [11]

    Unit3d: A unified trans- former for 3d dense captioning and visual grounding

    Zhenyu Chen, Ronghang Hu, Xinlei Chen, Matthias Nießner, and Angel X Chang. Unit3d: A unified trans- former for 3d dense captioning and visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18109–18119, 2023. 2, 6

    work page 2023

  12. [12]

    Back-tracing representative points for voting- based 3d object detection in point clouds

    Bowen Cheng, Lu Sheng, Shaoshuai Shi, Ming Yang, and Dong Xu. Back-tracing representative points for voting- based 3d object detection in point clouds. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8963–8972, 2021. 2

    work page 2021

  13. [13]

    4d spatio-temporal convnets: Minkowski convolutional neural networks, 2019

    Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks, 2019. 2, 3, 6, 7, 8, 11

    work page 2019

  14. [14]

    Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes, 2017. 5

    work page 2017

  15. [15]

    V otenet: A deep learning label fusion method for multi-atlas segmenta- tion

    Zhipeng Ding, Xu Han, and Marc Niethammer. V otenet: A deep learning label fusion method for multi-atlas segmenta- tion. InMedical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part III 22, pages 202–210. Springer, 2019. 2

    work page 2019

  16. [16]

    An empirical study of training end-to-end vision-and-language transformers

    Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and-language transformers. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18166–18176, 2022. 3

    work page 2022

  17. [17]

    Learning lightweight lane detection CNNs by self atten- tion distillation

    Yuenan Hou, Zheng Ma, Chunxiao Liu, and Chen Change Loy. Learning lightweight lane detection CNNs by self atten- tion distillation. InIEEE International Conference on Com- puter Vision, pages 1013–1021, 2019. 3

    work page 2019

  18. [18]

    Point-to-Voxel Knowledge Distillation for Li- DAR Semantic Segmentation

    Yuenan Hou, Xinge Zhu, Yuexin Ma, Chen Change Loy, and Yikang Li. Point-to-Voxel Knowledge Distillation for Li- DAR Semantic Segmentation. InIEEE Conference on Com- puter Vision and Pattern Recognition, pages 8479–8488,

    work page

  19. [19]

    Scaling up vision-language pre-training for image captioning

    Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17980–17989, 2022. 3

    work page 2022

  20. [20]

    Nerf-det++: Incorporat- ing semantic cues and perspective-aware depth supervision for indoor multi-view 3d detection.IEEE Transactions on Image Processing, 2025

    Chenxi Huang, Yuenan Hou, Weicai Ye, Di Huang, Xiaoshui Huang, Binbin Lin, and Deng Cai. Nerf-det++: Incorporat- ing semantic cues and perspective-aware depth supervision for indoor multi-view 3d detection.IEEE Transactions on Image Processing, 2025. 3

    work page 2025

  21. [21]

    Perturb, predict & para- phrase: Semi-supervised learning using noisy student for im- age captioning

    Arjit Jain, Pranay Reddy Samala, Preethi Jyothi, Deepak Mittal, and Maneesh Kumar Singh. Perturb, predict & para- phrase: Semi-supervised learning using noisy student for im- age captioning. InIJCAI, pages 758–764, 2021. 3

    work page 2021

  22. [22]

    Recurrent fusion network for image captioning

    Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, and Tong Zhang. Recurrent fusion network for image captioning. In Proceedings of the European conference on computer vision (ECCV), pages 499–515, 2018. 3

    work page 2018

  23. [23]

    More: Multi-order relation mining for dense captioning in 3d scenes

    Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. More: Multi-order relation mining for dense captioning in 3d scenes. InEuropean Conference on Computer Vision, pages 528–545. Springer, 2022. 1, 2, 6

    work page 2022

  24. [24]

    Context-aware alignment and mutual masking for 3d- language pre-training

    Zhao Jin, Munawar Hayat, Yuwei Yang, Yulan Guo, and Yin- jie Lei. Context-aware alignment and mutual masking for 3d- language pre-training. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 10984–10994, 2023. 2, 6 9

    work page 2023

  25. [25]

    DLIP: Distilling Language-Image Pre-training

    Huafeng Kuang, Jie Wu, Xiawu Zheng, Ming Li, Xue- feng Xiao, Rui Wang, Min Zheng, and Rongrong Ji. Dlip: Distilling language-image pre-training.arXiv preprint arXiv:2308.12956, 2023. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  26. [26]

    Align before fuse: Vision and language representation learn- ing with momentum distillation.Advances in neural infor- mation processing systems, 34:9694–9705, 2021

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation.Advances in neural infor- mation processing systems, 34:9694–9705, 2021. 3

    work page 2021

  27. [27]

    Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. InIn- ternational Conference on Machine Learning, pages 12888– 12900. PMLR, 2022. 3

    work page 2022

  28. [28]

    Oscar: Object-semantics aligned pre-training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16, pages 121–137. Springer,

    work page 2020

  29. [29]

    Moe3d: Mixture of ex- perts meets multi-modal 3d understanding.arXiv preprint arXiv:2511.22103, 2025

    Yu Li, Yuenan Hou, Yingmei Wei, Xinge Zhu, Yuexin Ma, Wenqi Shao, and Yanming Guo. Moe3d: Mixture of ex- perts meets multi-modal 3d understanding.arXiv preprint arXiv:2511.22103, 2025. 3

    work page arXiv 2025

  30. [30]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004. 5

    work page 2004

  31. [31]

    Complete 3d relationships extraction modality align- ment network for 3d dense captioning.IEEE Transactions on Visualization and Computer Graphics, 2023

    Aihua Mao, Zhi Yang, Wanxin Chen, Ran Yi, and Yong-jin Liu. Complete 3d relationships extraction modality align- ment network for 3d dense captioning.IEEE Transactions on Visualization and Computer Graphics, 2023. 2, 6

    work page 2023

  32. [32]

    An end-to- end transformer model for 3d object detection

    Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to- end transformer model for 3d object detection. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 2906–2917, 2021. 1, 7

    work page 2021

  33. [33]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318,

    work page

  34. [34]

    Pointnet: Deep learning on point sets for 3d classification and segmentation

    Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660,

    work page

  35. [35]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space.Advances in neural information processing systems, 30, 2017

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space.Advances in neural information processing systems, 30, 2017. 1

    work page 2017

  36. [36]

    Qi, Or Litany, Kaiming He, and Leonidas J

    Charles R. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas. Deep hough voting for 3d object detection in point clouds, 2019. 5

    work page 2019

  37. [37]

    Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel

    Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning, 2017. 6

    work page 2017

  38. [38]

    Octnet: Learning deep 3d representations at high resolutions

    Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3577–3586, 2017. 3

    work page 2017

  39. [39]

    Fcaf3d: Fully convolutional anchor-free 3d object detection

    Danila Rukhovich, Anna V orontsova, and Anton Konushin. Fcaf3d: Fully convolutional anchor-free 3d object detection. InEuropean Conference on Computer Vision, pages 477–

    work page

  40. [40]

    Springer, 2022. 2, 3, 4

    work page 2022

  41. [41]

    VL-BERT: Pre-training of Generic Visual-Linguistic Representations

    Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual- linguistic representations.arXiv preprint arXiv:1908.08530,

    work page internal anchor Pith review Pith/arXiv arXiv 1908

  42. [42]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion, 2015. 5

    work page 2015

  43. [43]

    Cagroup3d: Class- aware grouping for 3d object detection on point clouds

    Haiyang Wang, Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, Liwei Wang, et al. Cagroup3d: Class- aware grouping for 3d object detection on point clouds. Advances in Neural Information Processing Systems, 35: 29975–29988, 2022. 2, 3, 4, 5

    work page 2022

  44. [44]

    Rbgnet: Ray-based grouping for 3d object detection

    Haiyang Wang, Shaoshuai Shi, Ze Yang, Rongyao Fang, Qi Qian, Hongsheng Li, Bernt Schiele, and Liwei Wang. Rbgnet: Ray-based grouping for 3d object detection. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1110–1119, 2022. 2

    work page 2022

  45. [45]

    Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds

    Heng Wang, Chaoyi Zhang, Jianhui Yu, and Weidong Cai. Spatiality-guided transformer for 3d dense captioning on point clouds.arXiv preprint arXiv:2204.10688, 2022. 1, 2, 6

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  46. [46]

    Point transformer v2: Grouped vector atten- tion and partition-based pooling, 2022

    Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Heng- shuang Zhao. Point transformer v2: Grouped vector atten- tion and partition-based pooling, 2022. 2, 3, 7, 8

    work page 2022

  47. [47]

    Taseg: Temporal aggregation network for lidar semantic segmentation

    Xiaopei Wu, Yuenan Hou, Xiaoshui Huang, Binbin Lin, Tong He, Xinge Zhu, Yuexin Ma, Boxi Wu, Haifeng Liu, Deng Cai, et al. Taseg: Temporal aggregation network for lidar semantic segmentation. InIEEE Conference on Com- puter Vision and Pattern Recognition, pages 15311–15320,

    work page

  48. [48]

    Second: Sparsely embed- ded convolutional detection.Sensors, 18(10):3337, 2018

    Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embed- ded convolutional detection.Sensors, 18(10):3337, 2018. 2

    work page 2018

  49. [49]

    Vision-language pre-training with triple contrastive learning

    Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15671– 15680, 2022. 3

    work page 2022

  50. [50]

    Swin3d: A pretrained transformer backbone for 3d indoor scene understanding, 2023

    Yu-Qi Yang, Yu-Xiao Guo, Jian-Yu Xiong, Yang Liu, Hao Pan, Peng-Shuai Wang, Xin Tong, and Baining Guo. Swin3d: A pretrained transformer backbone for 3d indoor scene understanding, 2023. 2, 3, 6, 7, 8

    work page 2023

  51. [51]

    X-trans2cap: Cross- modal knowledge transfer using transformer for 3d dense captioning

    Zhihao Yuan, Xu Yan, Yinghong Liao, Yao Guo, Guan- bin Li, Shuguang Cui, and Zhen Li. X-trans2cap: Cross- modal knowledge transfer using transformer for 3d dense captioning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8563– 8573, 2022. 1, 6

    work page 2022

  52. [52]

    H3dnet: 3d object detection using hybrid geometric primi- tives

    Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primi- tives. InComputer Vision–ECCV 2020: 16th European Con- 10 ference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pages 311–329. Springer, 2020. 2

    work page 2020

  53. [53]

    Contextual modeling for 3d dense captioning on point clouds, 2022

    Yufeng Zhong, Long Xu, Jiebo Luo, and Lin Ma. Contextual modeling for 3d dense captioning on point clouds, 2022. 1, 6

    work page 2022

  54. [54]

    V oxelnet: End-to-end learning for point cloud based 3d object detection

    Yin Zhou and Oncel Tuzel. V oxelnet: End-to-end learning for point cloud based 3d object detection. InProceedings of the IEEE conference on computer vision and pattern recog- nition, pages 4490–4499, 2018. 2

    work page 2018

  55. [55]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911– 2921, 2023. 2, 6 A. Additional Ablations A.1. Ablation Study on Backbone Capacity In our V oxelCapNet, we find that the backbo...

    work page 2023

This paper was first reviewed by glm-5.2 on July 8, 2026.