REVIEW 29 cited by
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following
read the original abstract
We introduce Point-Bind, a 3D multi-modality model aligning point clouds with 2D image, language, audio, and video. Guided by ImageBind, we construct a joint embedding space between 3D and multi-modalities, enabling many promising applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D open-world understanding. On top of this, we further present Point-LLM, the first 3D large language model (LLM) following 3D multi-modal instructions. By parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction data, but exhibits superior 3D and multi-modal question-answering capacity. We hope our work may cast a light on the community for extending 3D point clouds to multi-modality applications. Code is available at https://github.com/ZiyuGuo99/Point-Bind_Point-LLM.
Forward citations
Cited by 29 Pith papers
-
3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
By inserting per-region markers and reserved vocabulary tokens before frozen encoder patches and refining them via MSR, 3D-PLOT-LLM adds part-level addressing to 3D LLMs, outperforming baselines on PartVerse-QA and 3D...
-
Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark
PolyMATH is a new 5,000-image benchmark where top MLLMs reach at most 41 percent accuracy on multi-modal mathematical reasoning, with ablation showing minimal gain from text over images.
-
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
LLaVA-NeXT-Interleave unifies multi-image, video, and 3D capabilities in large multimodal models via a new 1.18M-sample interleaved dataset and benchmark, achieving leading results across those tasks while preserving ...
-
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
MathVerse is a benchmark that tests multi-modal LLMs on visual math by providing each problem in six versions with progressively less diagram and text information to measure true visual understanding.
-
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
LLaMA-Adapter turns frozen LLaMA 7B into a capable instruction follower using only 1.2M new parameters and zero-init attention, matching Alpaca while extending to image-conditioned reasoning on ScienceQA and COCO.
-
Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding
ViPS fuses five complementary visual priors into an MLLM via lightweight distillation proxies and query-conditioned dynamic weighting, reporting state-of-the-art results on VSI-Bench and ScanNet-series benchmarks.
-
Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation
Hallo4D uses vision-language models to detect and correct spatial and temporal mistakes in AI-generated 3D and 4D content, improving consistency without retraining the base generators.
-
MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes
MV-GEL localizes fine-grained geometric entities on 3D meshes from natural language by ranking informative views with GELviews, applying VLM segmentation, and lifting masks via geometry-aware ray casting, reporting up...
-
MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes
MV-GEL uses a learned view selector and a fine-tuned vision-language segmentation model to localize text-described faces and edges on 3D meshes.
-
Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding
Occ-VLM reconstructs 3D occupancy from 2D images via a single encoder to ground vision-language reasoning, claiming SOTA occupancy prediction and parity with 3D-input VLMs on VQA and captioning.
-
RGB-Pointmap Pretraining for Unified 3D Scene Understanding
UniScene3D learns unified 3D scene representations from colored pointmaps using contrastive CLIP pretraining plus cross-view geometric and grounded view alignments, achieving state-of-the-art results on viewpoint grou...
-
Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM
Chat-Scene++ improves 3D scene understanding in multimodal LLMs by representing scenes as context-rich object sequences with identifier tokens and grounded chain-of-thought reasoning, reaching state-of-the-art on five...
-
Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding
Fusing intermediate-layer, mid-noise features of a video diffusion model with semantic tokens via learned gating improves MLLMs on 3D scene understanding benchmarks.
-
Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding
VEGA-3D extracts intermediate spatiotemporal features from a pretrained video diffusion model and fuses them into MLLMs to improve geometric and embodied reasoning without explicit 3D supervision.
-
Feeling the Space: Egomotion-Aware Video Representation for Efficient and Accurate 3D Scene Understanding
Motion-MLLM integrates IMU egomotion data into MLLMs using cascaded filtering and asymmetric fusion to ground visual content in physical trajectories for scale-aware 3D understanding, achieving competitive accuracy at...
-
Pointy - A Lightweight Transformer for Point Cloud Foundation Models
Pointy, a lightweight transformer trained on 39k point clouds, outperforms larger foundation models trained on 200k+ samples and nears SOTA from million-sample multimodal models.
-
Calibrated Multimodal Representation Learning with Missing Modalities
CalMRL mitigates anchor shift in multimodal representation learning by calibrating incomplete alignments through representation-level imputation of missing modalities using priors and a bi-step optimization with close...
-
SVL: Spike-based Vision-language Pretraining for Efficient 3D Open-world Understanding
SVL pretraining enables SNNs to reach 85.4% top-1 accuracy on zero-shot 3D classification while outperforming prior SNNs on detection, segmentation, and action recognition with added open-world QA capability.
-
Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation
Hallo4D mitigates 3D/4D generation hallucinations via LMM-based detection, multi-model voting correction, and motion-aware optimization without retraining base generators.
-
MathVis-Fine: Aligning Visual Supervision with Necessity via Progressive Dependency-Guided Training for Multimodal Mathematical Reasoning
MathVis-Fine proposes a dataset with fine-grained visual annotations and dependency ratings plus a progressive two-stage training paradigm to align visual supervision with sample-specific necessity in multimodal mathe...
-
SGSoft: Learning Fused Semantic-Geometric Features for 3D Shape Correspondence via Template-Guided Soft Signals
SGSoft introduces a template-guided pipeline that fuses semantic and geometric features to learn dense correspondences across deformable 3D shapes with claimed SOTA generalization and real-time efficiency.
-
Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities
Introduces MAF framework and DeepModal-Bench to capture universal cross-modal forgery traces for better generalization in multimodal deepfake detection.
-
Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMs
Efficient3D prunes visual tokens in 3D MLLMs via DVTIE and ATR modules, reporting better performance than unpruned baselines on Scan2Cap and other benchmarks.
-
RGB-Pointmap Pretraining for Unified 3D Scene Understanding
A CLIP-aligned transformer pretrained on multi-view RGB-Pointmap inputs with cross-view geometric and grounded view alignment yields unified 3D scene features that transfer to several scene-understanding tasks.
-
Rethinking Multimodal Few-Shot 3D Point Cloud Segmentation: From Fused Refinement to Decoupled Arbitration
Decoupling geometric and semantic feature pathways in a few-shot 3D segmentation model yields modest but consistent mIoU improvements over the fused-refinement baseline on S3DIS and ScanNet.
-
LLaVA-OneVision: Easy Visual Task Transfer
LLaVA-OneVision is the first single open LMM to simultaneously achieve strong performance in single-image, multi-image, and video scenarios with cross-scenario transfer capabilities.
-
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models
SPHINX improves multi-modal LLMs through joint mixing of weights, tasks, and visual embeddings from varied sources to achieve stronger alignment and multi-purpose capabilities.
-
Reinforcing 3D Understanding in Point-VLMs via Geometric Reward Credit Assignment
Geometric Reward Credit Assignment disentangles rewards to geometric tokens and adds reprojection consistency to boost 3D keypoint accuracy from 0.64 to 0.93 and bounding box IoU to 0.686 on a ShapeNetCore benchmark w...
-
SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video
SpaceEra++ adds ScenePick frame sampling and SpaceAlign pairwise constraints to the prior SpaceEra system, claiming consistent benchmark gains for 3D video spatial reasoning.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.