REVIEW 33 cited by
ConceptFusion: Open-set Multimodal 3D Mapping
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Building 3D maps of the environment is central to robot navigation, planning, and interaction with objects in a scene. Most existing approaches that integrate semantic concepts with 3D maps largely remain confined to the closed-set setting: they can only reason about a finite set of concepts, pre-defined at training time. Further, these maps can only be queried using class labels, or in recent work, using text prompts. We address both these issues with ConceptFusion, a scene representation that is (1) fundamentally open-set, enabling reasoning beyond a closed set of concepts and (ii) inherently multimodal, enabling a diverse range of possible queries to the 3D map, from language, to images, to audio, to 3D geometry, all working in concert. ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio. We demonstrate that pixel-aligned open-set features can be fused into 3D maps via traditional SLAM and multi-view fusion approaches. This enables effective zero-shot spatial reasoning, not needing any additional training or finetuning, and retains long-tailed concepts better than supervised approaches, outperforming them by more than 40% margin on 3D IoU. We extensively evaluate ConceptFusion on a number of real-world datasets, simulated home environments, a real-world tabletop manipulation task, and an autonomous driving platform. We showcase new avenues for blending foundation models with 3D open-set multimodal mapping. For more information, visit our project page https://concept-fusion.github.io or watch our 5-minute explainer video https://www.youtube.com/watch?v=rkXgws8fiDs
Forward citations
Cited by 33 Pith papers
-
Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs
All-Angles Bench, a 2,132-question benchmark across 90 real scenes, shows current MLLMs score around 60% on multi-view understanding while humans score 82%, with the largest gaps in camera pose estimation and cross-vi...
-
Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3D
LOCATE 3D localizes objects named in referring expressions directly from posed RGB-D frames, achieving 61.7% top-1 accuracy at IoU 25 on SR3D/NR3D/ScanRefer and 63.7% when trained with a new 130K-annotation dataset.
-
3DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answering
A feature-diversity-guided token compression pipeline for 3D VLMs keeps 94.7% of original QA accuracy at 128 tokens and runs 1.92x faster than uncompressed LLaVA-3D.
-
Memory for Attention: Language-Conditioned Re-Perception with a Vision--Language--Motion Map
Persistent map memory schedules budgeted re-perception better than memoryless VLM priors, with gain equal to Var(√λ), and language-conditioned VLMM needs both open-vocabulary relevance and per-instance dynamics.
-
Vision-Language-Motion Maps: An Open-Vocabulary, Uncertainty-Aware, Queryable Motion Attribute for 3D Scene Maps
VLMM is a 3D map representation where each object carries a fused, uncertainty-aware motion attribute (language-based movability prior + observed geometric motion) that makes motion queries such as 'what is moving' an...
-
Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Robotics
JITOMA keeps 3D scene graph nodes dormant until a natural-language query activates only task-relevant objects, bounding active graph size and captioning latency while preserving grounding accuracy.
-
CoSAG: Compact Semantic Anchor Gaussians via Training-Free Rate-Distortion Coding
Training-free closed-form lift plus spatially predictive entropy coding of Gaussian-to-anchor bindings yields sub-megabyte open-vocabulary 3D fields that match or beat prior accuracy.
-
MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes
MV-GEL uses a learned view selector and a fine-tuned vision-language segmentation model to localize text-described faces and edges on 3D meshes.
-
FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning
Joint factor-graph inference over LLM- and geometry-constrained functional edges yields higher recall and substantially lower calibration error than independent pairwise functional scene-graph methods.
-
VDAWorld: World Modelling via VLM-Directed Abstraction and Simulation
A vision-language model writes a simulation program—grounded by segmentation and 3D tools—that predicts physically plausible futures from an image and caption, outperforming video generators on modified benchmarks.
-
From reactive to cognitive: brain-inspired spatial intelligence for embodied agents
A brain-inspired navigation system stores landmarks, routes, and map-like voxel features in structured spatial memory and uses MLLM-powered retrieval to achieve strong results across object, instance, instruction, and...
-
Towards Open-Vocabulary Multimodal 3D Object Detection with Attributes
OVODA combines a 3DETR-style detector with a frozen foundation model to detect novel objects and attributes in 3D scenes, and the OVAD dataset adds spatial and motion attribute labels to nuScenes.
-
IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering
A benchmark that scores vision-language models by reconstructing the 3D scene behind an image as executable Blender code finds the models fail mainly on spatial precision, not tool usage.
-
CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity
CodeDiffuser uses vision-language-model-generated code to build 3D attention maps that condition a diffusion policy, improving success on ambiguous language manipulation tasks compared with end-to-end baselines.
-
FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understanding
FreeQ-Graph builds a semantically aligned 3D scene graph using CLIP, LLaVA, and an LLM, and reports strong zero-shot results on 3D visual grounding, segmentation, and scene graph prediction.
-
Satellites Reveal Mobility: A Commuting Origin-destination Flow Generator for Global Cities
Satellite imagery plus population is enough to generate commuting origin-destination flows that closely match models using detailed sociodemographic and point-of-interest data.
-
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
Embodied VideoAgent augments an LLM-based video agent with persistent object memory built from egocentric video, depth, and pose, plus VLM-based memory updates, reporting gains on Ego4D-VQ3D, OpenEQA, and EnvQA.
-
GAGS: Granularity-Aware Feature Distillation for Language Gaussian Splatting
GAGS achieves strong open-vocabulary 3D localization and segmentation by using depth-aware SAM prompting and an unsupervised granularity factor to distill consistent CLIP features into a single Gaussian feature field.
-
SLGaussian: Fast Language Gaussian Splatting in Sparse Views
SLGaussian builds a 3D semantic field from two photos in a single forward pass, stores CLIP features in a memory bank for fast open-vocabulary queries, and reports higher IoU than LangSplat and LERF on the LERF and 3D...
-
ChatSplat: 3D Conversational Gaussian Splatting
ChatSplat learns a 3D conversational field in Gaussian Splatting that supports object-, view-, and scene-level chat with an LLM at real-time speeds.
-
Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modeling
Diorama produces a structured, CAD-based 3D scene model from one RGB image using pretrained foundation models and staged layout optimization, with no end-to-end training.
-
3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning
3D-Mem represents explored and unexplored regions as compact snapshot images that a vision-language model can reason over, improving embodied question answering and lifelong navigation.
-
PickScan: Object discovery and reconstruction from handheld interactions
An interaction-guided, class-agnostic pipeline that discovers and reconstructs each manipulated object as a separate 3D model from egocentric RGB-D video.
-
OV-MAP: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots
OV-MAP projects 2D masks into 3D and uses mesh-area voting to create zero-shot, open-vocabulary 3D instance segmentation maps.
-
ForesightNav: Learning Scene Imagination for Efficient Exploration
ForesightNav uses a learned imagination module that predicts unseen occupancy and CLIP semantic features from partial maps, then selects navigation goals from the imagined map, improving simulated PointNav and ObjectN...
-
MR-COGraphs: Communication-efficient Multi-Robot Open-vocabulary Mapping System via 3D Scene Graphs
A multi-robot mapping system compresses 512-dimensional object semantics to 3 numbers for transmission, cutting map-sharing data by over 80 percent with comparable query accuracy.
-
BIP3D: Bridging 2D Images and 3D Perception for Embodied Intelligence
BIP3D is an image-centric 3D perception model that uses 2D foundation model features with explicit 3D position encoding to beat point-cloud-based methods on the EmbodiedScan detection and grounding benchmarks.
-
Details Matter for Indoor Open-vocabulary 3D Instance Segmentation
A carefully engineered pipeline of 2D grounding, 3D tracking, proposal merging, and Alpha-CLIP classification with a standardized similarity filter achieves state-of-the-art open-vocabulary 3D instance segmentation on...
-
OpenFusion++: An Open-vocabulary Real-time Scene Understanding System
OpenFusion++ upgrades the OpenFusion real-time 3D mapping system with confidence-based boundary refinement, area-weighted semantic caching, and a two-stage query matching that improves semantic accuracy and spatial qu...
-
Human-Robot Dialogue Annotation for Multi-Modal Common Ground
The authors present and release a multi-layer symbolic annotation of the SCOUT human-robot dialogue corpus, covering utterance semantics, multi-floor dialogue structure, and visual context for common-ground research.
-
Voxel-Aggregated Feature Synthesis: Efficient Dense Mapping for Simulated 3D Reasoning
VAFS replaces per-frame embedding in dense 3D mapping with per-object synthetic view embedding, using simulator ground-truth segmentation, achieving faster and higher-IoU semantic maps in simulation.
-
Gaussian Splatting as a Unified Representation for Autonomy in Unstructured Environments
Gaussian splatting is advocated as a unified navigation representation, but the experimental support is drawn entirely from the authors' earlier papers.
-
Multimodal Perception for Goal-oriented Navigation: A Survey
A literature survey that categorizes multimodal goal-oriented navigation methods into six inference domains and claims this taxonomy reveals cross-task computational patterns.
Discussion (0). Continue with ORCID to comment.