Pith. sign in

REVIEW 47 cited by

Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.11401 v2 pith:X3WBVDHP submitted 2024-03-18 cs.CV cs.AI

Rao Fu , Jingyu Liu , Xilun Chen , Yixin Nie , Wenhan Xiong This is my paper · ORCID
classification cs.CVcs.AI
keywords scene-llmmodelvisualego-centricfeaturesinformationinteractiveplanning
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

This paper introduces Scene-LLM, a 3D-visual-language model that enhances embodied agents' abilities in interactive 3D indoor environments by integrating the reasoning strengths of Large Language Models (LLMs). Scene-LLM adopts a hybrid 3D visual feature representation, that incorporates dense spatial information and supports scene state updates. The model employs a projection layer to efficiently project these features in the pre-trained textual embedding space, enabling effective interpretation of 3D visual information. Unique to our approach is the integration of both scene-level and ego-centric 3D information. This combination is pivotal for interactive planning, where scene-level data supports global planning and ego-centric data is important for localization. Notably, we use ego-centric 3D frame features for feature alignment, an efficient technique that enhances the model's ability to align features of small objects within the scene. Our experiments with Scene-LLM demonstrate its strong capabilities in dense captioning, question answering, and interactive planning. We believe Scene-LLM advances the field of 3D visual understanding and reasoning, offering new possibilities for sophisticated agent interactions in indoor settings.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 47 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hypo3D: Exploring Hypothetical Reasoning in 3D

    cs.CV 2025-02 conditional novelty 7.0 of 10

    Hypo3D, a 3D VQA benchmark with 14,885 QAs over 700 indoor scenes, tests whether models can reason about imagined scene changes and shows they fall far behind humans.

  2. GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models

    cs.CV 2025-01 conditional novelty 7.0 of 10

    Consistent object IDs across video frames and a reconstructed bird's-eye view let large VLMs, and a fine-tuned 7B model, reach state-of-the-art results on ScanNet 3D question answering, captioning, and grounding.

  3. GReFEM: Multimodal LLMs as Zero-Shot Semantic Assistants for Physics-Guided 3D Mesh Refinement

    cs.GR 2026-07 conditional novelty 6.5 of 10

    GReFEM shows MLLMs zero-shot isolate load-activated geometric features for volumetric mesh refinement with higher precision than matched-budget geometric heuristics.

  4. JOPP-3D: Joint Open Vocabulary Semantic Segmentation on Point Clouds and Panoramas

    cs.CV 2026-03 conditional novelty 6.5 of 10

    A training-free pipeline jointly segments panoramic images and reconstructed point clouds with open-vocabulary language queries via tangential decomposition, instance proposals, CLIP alignment, and depth-based label t...

  5. Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Qwen-3D feeds the full visual-language representation of a 3D scene into a mask decoder, beating prior 3D LMMs on grounding and segmentation benchmarks.

  6. Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    ViPS fuses five complementary visual priors into an MLLM via lightweight distillation proxies and query-conditioned dynamic weighting, reporting state-of-the-art results on VSI-Bench and ScanNet-series benchmarks.

  7. Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    SIS-Bench shows MLLMs lag on UAV self-awareness versus spatial tasks, and motion-aware optical-flow fusion improves perception and memory.

  8. Interleaved POMDP Planning for Multi-Object Search in Unknown Multi-Room Household Environments

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Interleaved high-level LLM-histogram POUCT and low-level particle-belief motion planning reduces collisions up to 63%, steps 35%, and detections 32% on multi-object search in unknown multi-room settings.

  9. Ego-Human Motion Prediction with 3D-Aware LLM

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Ego3DLM jointly predicts past and future 3D body pose and motion descriptions in a single autoregressive pass, conditioned on egocentric video, 3D scene features, and three-point tracking, achieving state-of-the-art o...

  10. ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    ELSA3D introduces elastic semantic anchoring via sparse anchor tokens and a scale-aware octree tokenizer to unify 3D generation and captioning at reduced computational cost.

  11. CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Topology-aware attention over hierarchical scene graphs lets a 3D-LLM ground, caption, and answer questions across multi-room homes, with large gains on a new HM3D benchmark.

  12. SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving

    cs.CV 2025-12 conditional novelty 6.0 of 10

    SpaceDrive replaces textual coordinate tokens with shared 3D positional encodings in a VLM driving planner, achieving state-of-the-art open-loop planning on nuScenes and 78.02 Driving Score on Bench2Drive.

  13. MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Mesh

    cs.GR 2025-08 unverdicted novelty 6.0 of 10

    MeshLLM improves LLM-based 3D mesh understanding and generation through primitive decomposition, a 1500k+ sample dataset, and topology-focused training strategies.

  14. City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning

    cs.CV 2025-07 reject novelty 6.0 of 10

    A new outdoor multiview multimodal QA dataset and a VAE-fused LVLM are presented with claims of large gains over prior models, but the evaluation is clouded by likely train-test source overlap and an undefined average.

  15. StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling

    cs.RO 2025-07 conditional novelty 6.0 of 10

    A streaming navigation framework combining a sliding-window KV cache with depth-based token pruning achieves state-of-the-art results on VLN-CE benchmarks with bounded context and low latency.

  16. Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Video-grounded RL with two-stage 2D grounding and SAM2 lifting enables 3D object localization and QA without dense 3D instance supervision.

  17. FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understanding

    cs.CV 2025-06 conditional novelty 6.0 of 10

    FreeQ-Graph builds a semantically aligned 3D scene graph using CLIP, LLaVA, and an LLM, and reports strong zero-shot results on 3D visual grounding, segmentation, and scene graph prediction.

  18. AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making

    cs.RO 2025-06 conditional novelty 6.0 of 10

    AntiGrounding lifts candidate robot trajectories into the VLM's visual space via multi-view rendering and structured VQA, and reports 57.5% average success across eight manipulation tasks, beating three intermediate-r...

  19. ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A training-free pipeline that generates editable 3D scenes from text by using a generated 2D image as an intermediary to extract object shapes, appearances, positions, and poses.

  20. Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Dynam3D represents scenes as patch, instance, and zone tokens that update dynamically, and feeds them to a 3.8B vision-language model to improve action prediction in vision-and-language navigation.

  21. Vision language models have difficulty recognizing virtual objects

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Vision language models frequently fail to include mentioned-but-not-depicted objects in their answers, revealing a limitation in virtual object tracking.

  22. SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    SpatialPrompting shows that keyframe selection plus camera-pose prompting can make a pre-trained multimodal LLM competitive on 3D question answering without 3D-specific fine-tuning.

  23. Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding

    cs.CV 2025-04 conditional novelty 6.0 of 10

    A contrastive pre-training method aligns 3D point features with language at the entity level and achieves state-of-the-art open-vocabulary semantic segmentation on ScanNet.

  24. Vision language models are unreliable at trivial spatial cognition

    cs.CV 2025-04 conditional novelty 6.0 of 10

    Three vision-language models give inconsistent left/right judgments on simple synthetic tabletop images when the same relation is probed with logically equivalent prompt variations.

  25. 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A multi-purpose Omni Superpoint Transformer lets a single 3D large multimodal model achieve state-of-the-art results on 3D question answering, dense captioning, and referring segmentation using point clouds only.

  26. 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A learnable 3D scene graph representation that injects semantic relation embeddings between objects improves LLM performance on 3D grounding, captioning, and question answering.

  27. Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech

    cs.CV 2024-12 conditional novelty 6.0 of 10

    M2SE-VTTS combines RGB, depth, and Gemini-generated scene captions with local and global attention to improve reverberation modeling in visual text-to-speech.

  28. SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A training-free 3D visual grounding system that renders query-aligned images and spatial text descriptions for a 2D vision-language model, beating prior zero-shot methods on ScanRefer and Nr3D.

  29. LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    LayoutVLM couples VLM-generated pose estimates and spatial relations with differentiable optimization to create physically plausible, instruction-aligned 3D layouts.

  30. LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences

    cs.CV 2024-12 conditional novelty 6.0 of 10

    LSceneLLM chooses task-relevant 3D regions via LLM attention, magnifies their details, and improves large-scene 3D question answering, planning, and captioning.

  31. Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Adding 3D coordinate encodings to sampled RGB-D video frames lets a video LLM beat prior 3D scene understanding models on five benchmarks.

  32. Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modeling

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Diorama produces a structured, CAD-based 3D scene model from one RGB image using pretrained foundation models and staged layout optimization, with no end-to-end training.

  33. g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A 3D-language feature field pretrained on about 5,000 indoor scenes with nearly a million text descriptions improves vision-language navigation, zero-shot object navigation, and situated question answering.

  34. ROOT: VLM based System for Indoor Scene Understanding and Beyond

    cs.CV 2024-11 conditional novelty 6.0 of 10

    ROOT combines GPT-4V, GroundingDINO, SAM, and DepthAnything with a fine-tuned SceneVLM to produce hierarchical indoor scene graphs and object distance estimates from a single RGB image.

  35. Vision-Language-Policy Model for Dynamic Robot Task Planning

    cs.RO 2025-12 conditional novelty 5.0 of 10

    A LoRA-fine-tuned Qwen2.5-VL model generates hierarchical, JSON-formatted robot policies from images and instructions, and updates them online when tasks change (success > 70% in real-world trials).

  36. Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    Fast3D prunes up to 90% of object-centric visual tokens in 3D MLLMs while preserving about 96.8% of original benchmark performance, using a trained attention predictor and adaptive layer-wise pruning.

  37. Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actions

    cs.RO 2025-06 conditional novelty 5.0 of 10

    Using robot actions and LLM visual reasoning, the system identifies deformability properties of unseen objects with up to 78.57% accuracy, which helps plan bin-packing at over 96% success after replanning.

  38. Zero-Shot 3D Visual Grounding from Vision-Language Models

    cs.CV 2025-05 conditional novelty 5.0 of 10

    SeeGround localizes objects in 3D scenes from natural language without 3D-specific training, using query-aligned rendered views and spatially enriched text fed to a 2D vision-language model.

  39. Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A sparse mixture-of-experts 3D multimodal LLM adaptively fuses RGB, RGBD, BEV, point cloud, and voxel tokens, achieving SOTA on several ScanNet-based 3D scene understanding benchmarks.

  40. RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A rule-guided spatial-aware network that localizes all mentioned entities in a 3D scene and uses target-position weak supervision raises ScanRefer 3D-RES mIoU from 39.5 to 44.6.

  41. PerLA: Perceptive 3D Language Assistant

    cs.CV 2024-11 conditional novelty 5.0 of 10

    PerLA improves 3D question answering and dense captioning by combining Hilbert-curve partitioned local point-cloud details with global context through cross-attention and graph convolution.

  42. RelMap: Reliable Spatiotemporal Sensor Data Visualization via Imputative Spatial Interpolation

    cs.LG 2025-08 unverdicted novelty 4.0 of 10

    RelMap combines GNN-based imputation with spatial interpolation and uncertainty-aware heatmaps for spatiotemporal sensor data.

  43. DreamScene: 3D Gaussian-based End-to-end Text-to-3D Scene Generation

    cs.CV 2025-07 conditional novelty 4.0 of 10

    A pipeline that generates editable 3D scenes from natural language by combining LLM-based layout planning, multi-timestep diffusion distillation, and staged camera sampling.

  44. AdaToken-3D: Dynamic Spatial Gating for Efficient 3D Large Multimodal-Models Reasoning

    cs.GR 2025-05 conditional novelty 4.0 of 10

    AdaToken-3D uses attention-derived contribution scores to prune spatial tokens layer by layer in 3D LMMs, achieving about 60 percent FLOPs reduction with roughly unchanged benchmark scores.

  45. Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering

    cs.CV 2025-02 conditional novelty 4.0 of 10

    A structured survey of 3D Scene Question Answering that categorizes datasets, methods, and metrics and finds a common encoder-fusion-prediction pipeline across approaches.

  46. CL3DOR: Contrastive Learning for 3D Large Multimodal Models via Odds Ratio on High-Resolution Point Clouds

    cs.CV 2025-01 conditional novelty 4.0 of 10

    CL3DOR pairs 8,192-point inputs, GPT-4o-generated hard-negative response triplets, and an odds-ratio contrastive loss to achieve state-of-the-art results on 3D scene understanding benchmarks.

  47. Scene Co-pilot: Procedural Text to Video Generation with Human in the Loop

    cs.CV 2024-11 conditional novelty 4.0 of 10

    Scene Copilot is a training-free pipeline that combines an LLM with retrieval over Infinigen's codebase and human-in-the-loop Blender editing to generate customized 3D scenes and videos from text prompts.

Pith tools