JM3D & JM3D-LLM: Elevating 3D Understanding with Joint Multi-modal Cues

Changli Wu; Haowei Wang; Jiayi Ji; Rongrong Ji; Xiaoshuai Sun; Yiwei Ma

RECORD

JM3D & JM3D-LLM: Elevating 3D Understanding with Joint Multi-modal Cues

Jiayi Ji, Haowei Wang, Changli Wu, Yiwei Ma, Xiaoshuai Sun, Rongrong Ji

Computer Vision and Pattern Recognition · Submitted 14 Oct 2023

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Evaluations on ModelNet40 and ScanObjectNN establish JM3D's superiority.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2310.09503 (version 3)

Pith IDU4YMOTUG, Pith's permanent record of this paper and its review

CitationChangli Wu, Haowei Wang, Jiayi Ji, Rongrong Ji, Xiaoshuai Sun, Yiwei Ma. "JM3D & JM3D-LLM: Elevating 3D Understanding with Joint Multi-modal Cues." arXiv:2310.09503 (2023). https://pith.science/paper/U4YMOTUG

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,642 input tokens.

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.10 · human participants 0.02 · controlled comparison 0.26 · randomized assignment 0.01 · states quantitative result 0.04 · compares with prior results 0.67 · mentions code or data availability 0.98.

The abstract, sentence by sentence

arXiv listing abstract v3. Results and the main claim are underlined; show every role.

The rising importance of 3D understanding, pivotal in computer vision, autonomous driving, and robotics, is evident. However, a prevailing trend, which straightforwardly resorted to transferring 2D alignment strategies to the 3D domain, encounters three distinct challenges: (1) Information Degradation: This arises from the alignment of 3D data with mere single-view 2D images and generic texts, neglecting the need for multi-view images and detailed subcategory texts. (2) Insufficient Synergy: These strategies align 3D representations to image and text features individually, hampering the overall optimization for 3D models. (3) Underutilization: The fine-grained information inherent in the learned representations is often not fully exploited, indicating a potential loss in detail. To address these issues, we introduce JM3D, a comprehensive approach integrating point cloud, text, and image. Key contributions include the Structured Multimodal Organizer (SMO), enriching vision-language representation with multiple views and hierarchical text, and the Joint Multi-modal Alignment (JMA), combining language understanding with visual representation. Our advanced model, JM3D-LLM, marries 3D representation with large language models via efficient fine-tuning. Evaluations on ModelNet40 and ScanObjectNN establish JM3D's superiority. The superior performance of JM3D-LLM further underscores the effectiveness of our representation transfer approach. Our code and models are available at https://github.com/Mr-Neko/JM3D.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	The rising importance of 3D understanding, pivotal in computer vision, autonomous driving, and robotics, is evident.	background 1.00	0.04	0.02	asserted 1.00
s1	However, a prevailing trend, which straightforwardly resorted to transferring 2D alignment strategies to the 3D domain, encounters three distinct challenges: (1) Information Degradation: This arises from the alignment of 3D data with mere single-view 2D images and generic texts, neglecting the need for multi-view images and detailed subcategory texts.	gap 0.69	0.05	0.03	asserted 1.00
s2	(2) Insufficient Synergy: These strategies align 3D representations to image and text features individually, hampering the overall optimization for 3D models.	gap 0.62	0.06	0.02	asserted 1.00
s3	(3) Underutilization: The fine-grained information inherent in the learned representations is often not fully exploited, indicating a potential loss in detail.	gap 0.68	0.12	0.02	hedged possibility 0.84
s4	To address these issues, we introduce JM3D, a comprehensive approach integrating point cloud, text, and image.	contribution 0.90	0.87	0.05	asserted 0.99
s5	Key contributions include the Structured Multimodal Organizer (SMO), enriching vision-language representation with multiple views and hierarchical text, and the Joint Multi-modal Alignment (JMA), combining language understanding with visual representation.	contribution 0.98	0.89	0.06	asserted 0.98
s6	Our advanced model, JM3D-LLM, marries 3D representation with large language models via efficient fine-tuning.	method 0.50	0.84	0.06	asserted 0.99
s7	Evaluations on ModelNet40 and ScanObjectNN establish JM3D's superiority.	result 1.00	0.93	0.34	asserted 1.00
s8	The superior performance of JM3D-LLM further underscores the effectiveness of our representation transfer approach.	interpretation 0.86	0.85	0.35	asserted 0.96
s9	Our code and models are available at https://github.com/Mr-Neko/JM3D.	contribution 0.68	0.48	0.02	asserted 0.97

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99.

What was examined, and what was not

sources read

arXiv listing abstract v3

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

The rising importance of 3D understanding, pivotal in computer vision, autonomous driving, and robotics, is evident. However, a prevailing trend, which straightforwardly resorted to transferring 2D alignment strategies to the 3D domain, encounters three distinct challenges: (1) Information Degradation: This arises from the alignment of 3D data with mere single-view 2D images and generic texts, neglecting the need for multi-view images and detailed subcategory texts. (2) Insufficient Synergy: These strategies align 3D representations to image and text features individually, hampering the overall optimization for 3D models. (3) Underutilization: The fine-grained information inherent in the learned representations is often not fully exploited, indicating a potential loss in detail. To address these issues, we introduce JM3D, a comprehensive approach integrating point cloud, text, and image. Key contributions include the Structured Multimodal Organizer (SMO), enriching vision-language representation with multiple views and hierarchical text, and the Joint Multi-modal Alignment (JMA), combining language understanding with visual representation. Our advanced model, JM3D-LLM, marries 3D representation with large language models via efficient fine-tuning. Evaluations on ModelNet40 and ScanObjectNN establish JM3D's superiority. The superior performance of JM3D-LLM further underscores the effectiveness of our representation transfer approach. Our code and models are available at https://github.com/Mr-Neko/JM3D.

Human review

How Pith checks papers → · Full audit of this paper →

References

How the literature uses it

3 arXiv papers cite this in 5 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers whose main result relates to this one's

Found among the papers closest to this one by title and main claim. Pith's classifier compared each pair's main results and a second model family confirmed the relation; checked against two further families, shown conflicts agreed 118 times in 120 and shown same results 218 times in 239. The quoted sentence is the other paper's main claim.

States the same result 1 paper

Extensive experiments on ModelNet40 and ScanObjectNN demonstrate the effectiveness of our proposed method, JM3D, which achieves state-of-the-art performance in zero-shot 3D classification.

Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation · 0.92

Papers citing it · 3 sentences without a confident label

Builds on it 1 paper

This approach is extended in JM3D-LLM [ref] , embedding 3D representations into LLMs for tasks such as image-3D retrieval and zero-shot 3D classification, providing more granular information.

Foundational Models for 3D Point Clouds: A Survey and Outlook · Triplet Alignment · 0.87

Cites it as background 1 paper

Large 3D point cloud-language models (3D-LLMs) introduce LLM into the point cloud modality [ref] .

MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors · Large 3D Point Cloud-Language Models · 0.99

What it cites · 16 sentences without a confident label

Uses the method or tool of 3 papers

Drawing from methodologies in recent studies [ref] , we first curated a comprehensive conversational dataset to fine-tune our model.

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models · Instruct Conversations for Point Querying · 0.93
Drawing from methodologies in recent studies [ref] , we first curated a comprehensive conversational dataset to fine-tune our model.

Visual Instruction Tuning · Instruct Conversations for Point Querying · 0.93
For instructive purposes, we adopt the methodology outlined in [ref] , devising 11 tailor-made templates tailored to the nuances of 3D modality.

LLaMA: Open and Efficient Foundation Language Models · Instruct Dataset · 0.98

Uses data or values from 2 papers

We source our instructional fine-tuning conversation data from Cap3D [ref] , which is a rich repository of point cloud samples curated from Objverse [ref] , accompanied by detailed descriptions.

Scalable 3D Captioning with Pretrained Models · Instruct Dataset · 1.00
We use ShapeNet55 [ref] as the pre-training dataset, which is the publicly-available subset of ShapeNet.

ShapeNet: An Information-Rich 3D Model Repository · Pretrain Dataset · 0.99

Compares with 1 paper

Remarkably, our JM3D model surpasses the former state-of-the-art ULIP [ref] across all 3D backbones, exhibiting superior top-1 accuracy.

ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding · Experiment Results · 1.00

Cites as support 2 papers

As proven by ULIP [ref] and CG3D [ref] , this approach, enriched by external information, markedly bolsters 3D understanding and showcases commendable transferability.

ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding · Introduction · 0.99
As proven by ULIP [ref] and CG3D [ref] , this approach, enriched by external information, markedly bolsters 3D understanding and showcases commendable transferability.

CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition · Introduction · 0.99

Cites as background 10 papers

ULIP [ref] explores the transfer ability of 2D contrastive learning to 3D by constructing a dataset combining point clouds, rendered images, and language descriptions from ShapeNet55 [ref] .

ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding · Preliminary · 0.84
ULIP [ref] explores the transfer ability of 2D contrastive learning to 3D by constructing a dataset combining point clouds, rendered images, and language descriptions from ShapeNet55 [ref] .

ShapeNet: An Information-Rich 3D Model Repository · Preliminary · 0.84
The landscape further evolved with multi-modal approaches like MiniGPT-4 [ref] , LLaVA [ref] , and LLaMA-adapter [ref] , leveraging LLM's inferential prowess to decipher visual content.

Visual Instruction Tuning · Large Language Model · 0.92

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.