Pith. sign in

REVIEW 3 major objections 4 minor 62 references

CL3DOR: Contrastive Learning for 3D Large Multimodal Models via Odds Ratio on High-Resolution Point Clouds

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CL3DOR claims that enriching point-cloud density to 8,192 points per object and adding GPT-4o-generated hard negatives trained with an odds-ratio contrastive loss makes a general-purpose 3D large multimodal model state of the art on…

desk verdict A solid empirical recipe for 3D LMMs whose component ablations hold up, but the headline SOTA claim is confounded by an unmatched LLM backbone and needs either a matched comparison or a softer claim. read the letter →

arxiv 2501.03879 v1 pith:GQTB2DDN submitted 2025-01-07 cs.CV cs.AI

Keonwoo Kim , Yeongjae Cho , Taebaek Hwang , Minsoo Jo , Sangdo Han This is my paper · ORCID
classification cs.CVcs.AI
keywords 3Dlargemultimodalmodelspointcloudunderstandingcontrastivelearningoddsratiolosshardnegativeresponsesinstructiontuningobjecthallucinationscene
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CL3DOR is a 3D large multimodal model that answers questions and writes captions about 3D scenes from point clouds. The paper argues that existing training data for such models is information-poor in two ways: point clouds are sampled too sparsely, and instruction responses come without informative wrong answers. To fix this, the authors sample 8,192 points per object, use GPT-4o to generate plausible-but-wrong hard negative responses, and add an odds-ratio contrastive term to the standard language-modeling loss. They report state-of-the-art results on four benchmarks, including a 5.3-point gain in refined exact-match on ScanQA and a 21-point CIDEr gain on Scan2Cap over the previous general-purpose model. The central bet is that richer visual input plus explicit penalization of near-miss answers yields finer spatial understanding and fewer object hallucinations.

What carries the argument

The load-bearing mechanism is the odds-ratio contrastive loss, adapted from preference optimization: $\mathrm{odds}_{\theta}(y\mid x)=P_{\theta}(y\mid x)/(1-P_{\theta}(y\mid x))$, with $\mathcal{L}_{\mathrm{OR}}=-\log\sigma\big(\log(\mathrm{odds}_{\theta}(y^+\mid x)/\mathrm{odds}_{\theta}(y^-\mid x))\big)$ added as an auxiliary term to negative log-likelihood. The odds ratio matters because it suppresses hard negatives without crushing their logits, which the authors argue is more stable than a raw probability-ratio loss. Around this loss sits a three-stage training pipeline: object alignment on single-object captions, scene alignment on referring-expression and scene-caption data, then spatial contrastive instruction tuning on triplet data. The visual side uses a frozen PointBERT encoder consuming 8,192 points per object, a spatial transformer for inter-object relations, and special point tokens injected into a LLaMA3-8B-Instruct backbone.

What would settle it

Retrain CL3DOR while holding the LLM backbone identical to the previous general-purpose baseline (and retrain that baseline with LLaMA3-8B-Instruct), keeping point-cloud resolution, hard negatives, and the OR loss exactly the same; if the 5.3-point ScanQA EM gain and 21-point CIDEr gain collapse, the proposed components are not the cause. Separately, have human annotators judge a random sample of GPT-4o-generated negatives for correctness and hardness; if many are wrong or easy, the negative-quality premise fails.

Watch

Extended reading notes

Core claim

The discovery claimed is that a single general-purpose 3D LMM can beat task-specific and generalist models on four benchmarks by combining three ingredients: (1) sampling 8,192 points per object instead of the usual 1,024; (2) augmenting instruction data with hard negatives that are plausible but swap location, color, count, or existence; and (3) training with $L = \mathcal{L}_{\mathrm{NLL}} + \lambda \mathcal{L}_{\mathrm{OR}}$, where $\mathcal{L}_{\mathrm{OR}} = -\log \sigma\big(\log \frac{\mathrm{odds}_{\theta}(y^+\mid x)}{\mathrm{odds}_{\theta}(y^-\mid x)}\big)$ and the odds of a response are its generation probability divided by one minus that probability. Ablations within the paper show each ingredient matters: dropping to low resolution lowers Scan2Cap CIDEr from 93.4 to 71.7, easy negatives underperform hard negatives, and removing the odds-ratio term by setting $\lambda=0$ reduces ScanQA CIDEr by roughly 10 points. The same model keeps its yes-rate close to the true object-existence rate on 3D-POPE, indicating less bias toward hallucinating objects than baselines.

Load-bearing premise

The main gain over the previous general-purpose model is measured without keeping the language-model backbone fixed, so a stronger newer LLM rather than the 8,192-point inputs, hard negatives, or odds-ratio loss could explain part of the improvement.

Editorial extensions

If this is right

  • High-resolution point clouds are nearly free: the paper reports the point encoder is only 0.27% of the model's parameters, so moving from 1,024 to 8,192 points raises fidelity without a meaningful compute penalty.
  • Hard negatives are doing real work beyond mere data augmentation: the hard-negative setting beats an easy-negative triplet setting on almost every metric, and the log-odds and reward-margin curves show hard negatives keep the model challenged throughout training.
  • The odds-ratio loss is stable to its weight: performance stays high for lambda from 5e-2 to 5e-1 and beats both plain SFT and the probability-ratio variant on ScanQA, SQA3D, and Scan2Cap.
  • Object hallucination is reduced as a by-product: on 3D-POPE, CL3DOR's yes-rate is far closer to the balanced baseline than prior 3D LMMs, which were heavily biased toward saying objects exist.
  • A single general-purpose model can beat task-specific and task-adaptive models on 3D captioning and QA without fine-tuning per benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's headline comparison is not backbone-matched, and the authors do not ablate the language model; a rerun that swaps LLaMA3-8B-Instruct for the exact backbone used by the previous general-purpose baseline would isolate how much of the gain comes from the three proposed components rather than from a newer LLM.
  • The GPT-4o-generated hard negatives are not human-validated; if a share of them are wrong or only trivially different, the measured contrastive benefit could partly reflect dataset filtering or noise, so a small human audit of negative quality would tighten the claim.
  • Because the odds-ratio loss is not specific to point clouds, the same triplet-construction and OR training recipe could transfer to 2D vision-language models or any generative instruction-following setting where near-miss hallucinations are the dominant failure mode.
  • The paper's own limitation note says resources allowed only a few epochs and prevented million-scale training; whether the reported SOTA gaps persist under longer training and larger data remains untested by the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CL3DOR, a 3D large multimodal model that combines three ingredients: (i) higher-resolution point clouds (8,192 points per object instead of the common 1,024), (ii) training triplets with GPT-4o-generated hard negative responses, and (iii) an odds-ratio (ORPO) auxiliary loss added to the standard negative log-likelihood loss during instruction tuning. The model is trained in three stages and evaluated on ScanQA, SQA3D, Scan2Cap, and 3D-POPE. The authors report state-of-the-art results on these benchmarks compared with single-task, task-adaptive, and general-purpose baselines, particularly LEO, and support their components with within-model ablations.

Significance. If the gains prove robust, the paper contributes a useful and simple recipe: high-resolution point cloud sampling and contrastive instruction tuning with hard negatives are modifications that could be adopted by other 3D LMMs. The within-model ablations (Table 4, Figure 5, Appendix Tables 6-9) are carefully designed and show consistent improvements from each component, and the authors commit to releasing the constructed datasets, which would benefit the community. However, the headline state-of-the-art claim is currently not established because the main comparison is confounded by the LLM backbone, and the evaluations lack error bars.

major comments (3)
  1. [§4, Table 2] The state-of-the-art claim is supported by a comparison against LEO in which the LLM backbone is not controlled. CL3DOR uses LLaMA3-8B-Instruct, as stated in Section 4 (Implementation details), while LEO's language model is not specified in the paper. The within-model ablations (Tables 4, 6-9) all fix LLaMA3, so they do not resolve this confound. To attribute the reported gains (e.g., 5.3 pp refined EM@1 on ScanQA, 21 CIDEr on Scan2Cap) to the proposed components, a backbone-matched comparison is needed.
  2. [Appendix A.2] The statement that all evaluations are conducted using a single inference run means the numbers in Tables 2-4 and 6-9 lack error bars. Given that many differences are small (e.g., 3D-POPE accuracy differences around 2-5 points between settings, or λ ablation differences of ~1-2 points), the paper should report variance across at least a few seeds or explicitly refrain from making fine-grained claims. At minimum, the main text should draw attention to this limitation, not only the appendix.
  3. [§3.3, Appendix A.3] The quality and difficulty of the GPT-4o-generated negatives are not validated. The prompt includes filters ("REMOVE THIS SAMPLE..."), but no statistics on the filter rate or a human evaluation of the remaining negatives are provided. If many negatives are mislabeled or not actually hard, the contrastive term could be learning from noise. Since these datasets are claimed as a contribution, the paper should include a validation study (e.g., human agreement or at least a sample of checked examples).
minor comments (4)
  1. [Section 5.2 and Figure 6] The text says hard negatives are shown with a green line and easy negatives with a blue line, but the Figure 6 caption states the opposite. The color assignment should be corrected to be consistent.
  2. [Section 5.2, Appendix A.4] The text refers to a "Discussion section" that does not exist in the manuscript; these references should point to the relevant sections (e.g., "Experimental Results" or "Appendix").
  3. [Table 2] There are minor formatting issues in baseline names: "V ote2Cap-DETR" has a stray space, and "V oteNet+MCAN" appears with inconsistent spacing in the text.
  4. [Limitation section] The limitation section is only one sentence and does not mention the LLM backbone confound or the single-run evaluation; consider expanding it to include these caveats.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CL3DOR's components are validated against external benchmarks, and the odds-ratio loss is imported from prior work rather than derived from the claimed results.

full rationale

The paper's derivation chain is not circular. The three proposed components — 8,192-point object sampling, GPT-4o-generated hard negatives, and the odds-ratio auxiliary loss — are each evaluated on external benchmarks (ScanQA, SQA3D, Scan2Cap, 3D-POPE) with fixed ground-truth labels, and the ablations in Tables 4-9 and Figure 5 compare variants on those same external metrics. The odds-ratio loss (Eqs. 1-3) is explicitly taken from Hong et al. (2024) and is not derived in this paper; the claim that OR improves over PR and SFT is an empirical comparison, not an identity. The high-resolution input is a training-data and architecture choice, and Table 4 shows its effect against external test sets rather than against quantities defined by the method itself. The hard-negative construction (Appendix Tables 11-12) is a data-generation procedure; training on these negatives does not define the benchmark scores, and the fact that negatives encode GPT-4o's plausibility judgments is a data-quality concern, not a circular reduction. The only overlapping self-citation is Byun et al. (2022), which includes author T. Hwang, and it is cited only as general inspiration for hard negatives; it is not load-bearing. The backbone mismatch between CL3DOR (LLaMA3-8B-Instruct) and LEO is a legitimate comparability threat for the SOTA claim, but that is a confound and correctness risk, not a circularity.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on two main free choices (lambda and point density), one underspecified data-generation rule, and several domain assumptions about transferable pretrained components, GPT-4o generated negatives, and metric validity. No new physical or mathematical entities are introduced.

free parameters (3)
  • lambda_OR (odds-ratio loss weight) = 3e-1 (chosen via ablation over 0, 5e-2, 1e-1, 3e-1, 5e-1)
    Main results use lambda=0.3; Figure 5 and Appendix Tables 6-9 show performance varies with lambda, and lambda=0 (pure SFT) drops notably. The value is selected on the evaluation benchmarks.
  • point samples per object = 8192
    Chosen to match PointBERT's pretraining resolution. The ablation only compares 1024 vs 8192, so the exact optimum and the scaling curve are not established.
  • object-existence negative sampling thresholds = unspecified
    Section 3.3 says 'no' responses are generated 'by considering the frequency and co-occurrence of objects in the entire dataset', but no thresholds, frequency cutoffs, or algorithm are given. This is a hidden hand-chosen setting that affects the hallucination results.
assumptions (6)
  • domain assumption PointBERT pretrained on ShapeNet at 8,192 points transfers to ScanNet scenes and remains frozen throughout training.
    Invoked in Appendix A.1: 'PointBERT remains frozen throughout all three training stages of CL3DOR, meaning its weights are not updated, thus preserving its pre-trained knowledge.'
  • domain assumption LLaMA3-8B-Instruct is an appropriate and sufficient language backbone for 3D instruction tuning.
    Invoked in Section 4 Implementation details: 'We employ LLaMA3-8B-Instruct (Dubey et al., 2024) as the LLM.' The paper does not control for backbone in the headline comparison with LEO.
  • domain assumption GPT-4o-generated hard negatives are plausible, correct as negatives, and genuinely hard for the target model.
    Invoked in Section 3.3 and Appendix Tables 11-12. Negative quality is not validated by human raters or by measuring the model's error distribution before generation.
  • domain assumption The pairwise spatial relation transformer captures the inter-object spatial relationships needed for 3D QA and captioning.
    Borrowed from Chen et al. (2022) and used in Section 3.1 without modification or re-validation in the 3D LMM setting.
  • standard math Standard autoregressive language-modeling factorization applies to point-cloud-conditioned generation.
    Used in Section 3.2 to define P_theta(y | x) and the NLL loss; this is standard probability factorization.
  • domain assumption Benchmark metrics (CIDEr, sentence similarity, refined exact match) reflect meaningful 3D understanding quality.
    Invoked in Section 4 Evaluation Metrics; the paper itself discounts BLEU-4 and METEOR on Scan2Cap, so choosing which metrics count as evidence is interpretive.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CL3DOR: Contrastive Learning for 3D Large Multimodal Models via Odds Ratio on High-Resolution Point Clouds." pith.science (2026). https://pith.science/paper/GQTB2DDN

@misc{pith2026250103879,
  author       = {Pith},
  title        = {Pith review of: CL3DOR: Contrastive Learning for 3D Large Multimodal Models via Odds Ratio on High-Resolution Point Clouds},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GQTB2DDN}},
  note         = {Machine review of arXiv:2501.03879}
}
read the original abstract

Recent research has demonstrated that Large Language Models (LLMs) are not limited to text-only tasks but can also function as multimodal models across various modalities, including audio, images, and videos. In particular, research on 3D Large Multimodal Models (3D LMMs) is making notable strides, driven by the potential of processing higher-dimensional data like point clouds. However, upon closer examination, we find that the visual and textual content within each sample of existing training datasets lacks both high informational granularity and clarity, which serve as a bottleneck for precise cross-modal understanding. To address these issues, we propose CL3DOR, Contrastive Learning for 3D large multimodal models via Odds ratio on high-Resolution point clouds, designed to ensure greater specificity and clarity in both visual and textual content. Specifically, we increase the density of point clouds per object and construct informative hard negative responses in the training dataset to penalize unwanted responses. To leverage hard negative responses, we incorporate the odds ratio as an auxiliary term for contrastive learning into the conventional language modeling loss. CL3DOR achieves state-of-the-art performance in 3D scene understanding and reasoning benchmarks. Additionally, we demonstrate the effectiveness of CL3DOR's key components through extensive experiments.

Figures

Figures reproduced from arXiv: 2501.03879 by the authors.

Figure 1
Figure 1. Comparison of training methods for 3D LMMs. The upper side shows existing 3D LMMs trained with sparse (low￾resolution) point clouds and positive labels, aiming to maxi￾mize logits for correct responses. The lower side illustrates the CL3DOR, which uses dense (high-resolution) point cloud input and incorporates both positive and negative labels, employing a contrastive learning approach to explicitly leverage negativ… view at source ↗
Figure 2
Figure 2. (a) Examples of the three-stage training datasets used in CL3DOR. Notably, only Stage 3 includes two types of responses following an instruction for contrastive learning. (b) The process of generating hard negative responses for the 3D question answering task. We use GPT-4o to create plausible hard negatives, referencing a top-view image and scene objects 2023; Sharkawy & Koustoumpardis, 2022). However, we find that… view at source ↗
Figure 3
Figure 3. Illustration of the proposed CL3DOR. The figure visualizes spatial contrastive instruction tuning. The objective function incorporates an odds ratio loss as an auxiliary term alongside the commonly used NLL loss for language modeling. 3.2. Training Paradigm in CL3DOR The training process of CL3DOR follows a three-stage paradigm, as outlined in (Huang et al., 2023). We include two alignment tuning stages, beginning w… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Resolution in point clouds: high-resolution (left) with 8,192 point clouds per object, and low-resolution (right) with 1,024 point clouds per object. LOR = − log σ  log  oddsθ(y+ | x) oddsθ(y- | x)  (2) L = E(x,y+,y−) h LNLL + λ · LORi (3) 3.3. Dataset Refinement f…
Figure 5
Figure 5. Figure 5: Impact of λ on Performance. Plots (a) and (b) illustrate the CIDEr and EM@1-refined scores for ScanQA, while plots (c) and (d) depict the CIDEr and sentence similarity (Sim) scores for Scan2Cap across varying λ values. These results emphasize the crucial role of the OR…
Figure 6
Figure 6. Figure 6: Visualization of the log odds ratio (left) and the reward margin (right) for both hard negative and easy negative dataset settings. The blue line shows results using hard negatives, while the green line shows those using easy negatives. tuning: NLL loss with either the…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 21 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Achlioptas, P., Abdelreheem, A., Xia, F., Elhoseiny, M., and Guibas, L. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part I 16, pp.\ 422--440. Springer, 2020

    2020

  3. [3]

    Cont: Contrastive neural text generation

    An, C., Feng, J., Lv, K., Kong, L., Qiu, X., and Huang, X. Cont: Contrastive neural text generation. Advances in Neural Information Processing Systems, 35: 0 2197--2210, 2022

    work page 2022

  4. [4]

    Scanqa: 3d question answering for spatial scene understanding

    Azuma, D., Miyanishi, T., Kurita, S., and Kawanabe, M. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 19129--19139, 2022

    2022

  5. [5]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  6. [6]

    and Lavie, A

    Banerjee, S. and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp.\ 65--72, 2005

    2005

  7. [7]

    Grit-vlp: Grouped mini-batch sampling for efficient vision and language pre-training

    Byun, J., Hwang, T., Fu, J., and Moon, T. Grit-vlp: Grouped mini-batch sampling for efficient vision and language pre-training. In European Conference on Computer Vision, pp.\ 395--412. Springer, 2022

    work page 2022

  8. [8]

    3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds

    Cai, D., Zhao, L., Zhang, J., Sheng, L., and Xu, D. 3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 16464--16473, 2022

    work page 2022

Show all 62 references
  1. [9]

    Honeybee: Locality-enhanced projector for multimodal llm

    Cha, J., Kang, W., Mun, J., and Roh, B. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13817--13827, 2024

    2024

  2. [10]

    X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al

    Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015

    2015 arXiv

  3. [11]

    Your vision-language model itself is a strong filter: Towards high-quality instruction tuning with data selection

    Chen, R., Wu, Y., Chen, L., Liu, G., He, Q., Xiong, T., Liu, C., Guo, J., and Huang, H. Your vision-language model itself is a strong filter: Towards high-quality instruction tuning with data selection. arXiv preprint arXiv:2402.12501, 2024 a

    2024 arXiv

  4. [12]

    Language conditioned spatial relation reasoning for 3d object grounding

    Chen, S., Guhur, P.-L., Tapaswi, M., Schmid, C., and Laptev, I. Language conditioned spatial relation reasoning for 3d object grounding. Advances in neural information processing systems, 35: 0 20522--20535, 2022

    2022

  5. [13]

    End-to-end 3d dense captioning with vote2cap-detr

    Chen, S., Zhu, H., Chen, X., Lei, Y., Yu, G., and Chen, T. End-to-end 3d dense captioning with vote2cap-detr. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11124--11133, 2023

    2023

  6. [14]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning

    Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., and Chen, T. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 2642...

    2024

  7. [15]

    Chen, Z., Gholami, A., Nie ner, M., and Chang, A. X. Scan2cap: Context-aware dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 3193--3203, 2021

    2021

  8. [16]

    X., Savva, M., Halber, M., Funkhouser, T., and Nie ner, M

    Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nie ner, M. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 5828--5839, 2017

    2017

  9. [17]

    Objaverse: A universe of annotated 3d objects

    Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., and Farhadi, A. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13142-...

    2023

  10. [18]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  11. [19]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Fu, R., Liu, J., Chen, X., Nie, Y., and Xiong, W. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024

    2024 arXiv

  12. [20]

    Detecting and preventing hallucinations in large vision language models

    Gunjal, A., Yin, J., and Bas, E. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 18135--18143, 2024

    2024

  13. [21]

    Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following

    Guo, Z., Zhang, R., Zhu, X., Tang, Y., Ma, X., Han, J., Chen, K., Gao, P., Li, X., Li, H., et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.00615, 2023

    2023 arXiv

  14. [22]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  15. [23]

    Reference-free monolithic preference optimization with odds ratio

    Hong, J., Lee, N., and Thorne, J. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691, 2024

    2024 arXiv

  16. [24]

    3d-llm: Injecting the 3d world into large language models

    Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., and Gan, C. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36: 0 20482--20494, 2023

    2023

  17. [25]

    Chat-3d v2: Bridging 3d scene and large language models with object identifiers

    Huang, H., Wang, Z., Huang, R., Liu, L., Cheng, X., Zhao, Y., Jin, T., and Zhao, Z. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint arXiv:2312.08168, 2023

    2023 arXiv

  18. [26]

    An embodied generalist agent in 3d world

    Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y., Li, Q., Zhu, S.-C., Jia, B., and Huang, S. An embodied generalist agent in 3d world. In Proceedings of the International Conference on Machine Learning (ICML), 2024

    2024

  19. [27]

    U., Wang, Z., Nan, F., Li, X., Tan, M., Nallapati, R., Ray, B., Bhatia, P., et al

    Jain, N., Zhang, D., Ahmad, W. U., Wang, Z., Nan, F., Li, X., Tan, M., Nallapati, R., Ray, B., Bhatia, P., et al. Contraclm: Contrastive learning for causal language model. arXiv preprint arXiv:2210.01185, 2022

    2022 arXiv

  20. [28]

    Bootstrapping vision-language learning with decoupled language pre-training

    Jian, Y., Gao, C., and Vosoughi, S. Bootstrapping vision-language learning with decoupled language pre-training. Advances in Neural Information Processing Systems, 36, 2024

    2024

  21. [29]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  22. [30]

    L., Bansal, M., and Liu, J

    Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T. L., Bansal, M., and Liu, J. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 7331--7341, 2021

    2021

  23. [31]

    Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34: 0 9694--9705, 2021

    2021

  24. [32]

    Rouge: A package for automatic evaluation of summaries

    Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.\ 74--81, 2004

    2004

  25. [33]

    Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26296--26306, 2024 a

    2024

  26. [34]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024 b . URL https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  27. [35]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024 c

    2024

  28. [36]

    Brio: Bringing order to abstractive summarization

    Liu, Y., Liu, P., Radev, D., and Neubig, G. Brio: Bringing order to abstractive summarization. arXiv preprint arXiv:2203.16804, 2022

    2022 arXiv

  29. [37]

    Scalable 3d captioning with pretrained models

    Luo, T., Rockwell, C., Lee, H., and Johnson, J. Scalable 3d captioning with pretrained models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  30. [38]

    Sqa3d: Situated question answering in 3d scenes

    Ma, X., Yong, S., Zheng, Z., Li, Q., Liang, Y., Zhu, S.-C., and Huang, S. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022

    2022 arXiv

  31. [39]

    OpenAI. Gpt-4o. https://openai.com/index/hello-gpt-4o/, 2024

    2024

  32. [40]

    Bleu: a method for automatic evaluation of machine translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.\ 311--318, 2002

    2002

  33. [41]

    Advances and perspectives in collaborative robotics: a review of key technologies and emerging trends

    Patil, S., Vasu, V., and Srinadh, K. Advances and perspectives in collaborative robotics: a review of key technologies and emerging trends. Discover Mechanical Engineering, 2 0 (1): 0 13, 2023

    2023

  34. [42]

    Gpt4point: A unified framework for point-language understanding and generation

    Qi, Z., Fang, Y., Sun, Z., Wu, X., Wu, T., Wang, J., Lin, D., and Zhao, H. Gpt4point: A unified framework for point-language understanding and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26417--26427, 2024

    2024

  35. [43]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  36. [44]

    and Gurevych, I

    Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  37. [45]

    Contrastive learning with hard negative samples

    Robinson, J., Chuang, C.-Y., Sra, S., and Jegelka, S. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020

    2010 arXiv

  38. [46]

    Language-grounded indoor 3d semantic segmentation in the wild

    Rozenberszki, D., Litany, O., and Dai, A. Language-grounded indoor 3d semantic segmentation in the wild. In European Conference on Computer Vision, pp.\ 125--141. Springer, 2022

    2022

  39. [47]

    \"O ., and Pfister, T

    Sarkar, P., Ebrahimi, S., Etemad, A., Beirami, A., Ar k, S. \"O ., and Pfister, T. Mitigating object hallucination via data augmented contrastive tuning. arXiv preprint arXiv:2405.18654, 2024

    2024 arXiv

  40. [48]

    and Koustoumpardis, P

    Sharkawy, A.-N. and Koustoumpardis, P. N. Human--robot interaction: A review and analysis on variable admittance control, safety, and perspectives. Machines, 10 0 (7): 0 591, 2022

    2022

  41. [49]

    S., Love, J., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi \`e re, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  42. [50]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  43. [51]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  44. [52]

    Cider: Consensus-based image description evaluation

    Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 4566--4575, 2015

    2015

  45. [53]

    Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes

    Wang, Z., Huang, H., Zhao, Y., Zhang, Z., and Zhao, Z. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08769, 2023

    2023 arXiv

  46. [54]

    Pointllm: Empowering large language models to understand point clouds

    Xu, R., Wang, X., Wang, T., Chen, Y., Pang, J., and Lin, D. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308.16911, 2023

    2023 arXiv

  47. [55]

    Y., Zhou, W., Yin, F., Galstyan, A., Yin, W., and Chen, M

    Yan, T., Wang, F., Huang, J. Y., Zhou, W., Yin, F., Galstyan, A., Yin, W., and Chen, M. Contrastive instruction tuning. arXiv preprint arXiv:2402.11138, 2024

    2024 arXiv

  48. [56]

    F., and Chai, J

    Yang, J., Chen, X., Madaan, N., Iyengar, M., Qian, S., Fouhey, D. F., and Chai, J. 3d-grand: A million-scale dataset for 3d-llms with better grounding and less hallucination. arXiv preprint arXiv:2406.05132, 2024

    2024 arXiv

  49. [57]

    Point-bert: Pre-training 3d point cloud transformers with masked point modeling

    Yu, X., Tang, L., Rao, Y., Huang, T., Zhou, J., and Lu, J. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 19313--19322, 2022

    2022

  50. [58]

    Beyond llava-hd: Diving into high-resolution large multimodal models

    Zhang, Y.-F., Wen, Q., Fu, C., Wang, X., Zhang, Z., Wang, L., and Jin, R. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.08487, 2024

    2024 arXiv

  51. [59]

    Click: Controllable text generation with sequence likelihood contrastive learning

    Zheng, C., Ke, P., Zhang, Z., and Huang, M. Click: Controllable text generation with sequence likelihood contrastive learning. In Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 1022--1040, 2023

    2023

  52. [60]

    Lima: Less is more for alignment

    Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024 a

    2024

  53. [61]

    Uni3d: Exploring unified 3d representation at scale

    Zhou, J., Wang, J., Ma, B., Liu, Y.-S., Huang, T., and Wang, X. Uni3d: Exploring unified 3d representation at scale. In International Conference on Learning Representations (ICLR), 2024 b

    2024

  54. [62]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Zhu, Z., Ma, X., Chen, Y., Deng, Z., Huang, S., and Li, Q. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 2911--2921, 2023

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.