REVIEW 23 cited by
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
3D scene graphs have recently emerged as a powerful high-level representation of 3D environments. A 3D scene graph describes the environment as a layered graph where nodes represent spatial concepts at multiple levels of abstraction and edges represent relations between concepts. While 3D scene graphs can serve as an advanced "mental model" for robots, how to build such a rich representation in real-time is still uncharted territory. This paper describes a real-time Spatial Perception System, a suite of algorithms to build a 3D scene graph from sensor data in real-time. Our first contribution is to develop real-time algorithms to incrementally construct the layers of a scene graph as the robot explores the environment; these algorithms build a local Euclidean Signed Distance Function (ESDF) around the current robot location, extract a topological map of places from the ESDF, and then segment the places into rooms using an approach inspired by community-detection techniques. Our second contribution is to investigate loop closure detection and optimization in 3D scene graphs. We show that 3D scene graphs allow defining hierarchical descriptors for loop closure detection; our descriptors capture statistics across layers in the scene graph, ranging from low-level visual appearance to summary statistics about objects and places. We then propose the first algorithm to optimize a 3D scene graph in response to loop closures; our approach relies on embedded deformation graphs to simultaneously correct all layers of the scene graph. We implement the proposed Spatial Perception System into a architecture named Hydra, that combines fast early and mid-level perception processes with slower high-level perception. We evaluate Hydra on simulated and real data and show it is able to reconstruct 3D scene graphs with an accuracy comparable with batch offline methods despite running online.
Forward citations
Cited by 23 Pith papers
-
Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Models
VLMs excel at semantic and grouping tasks while VGMs are stronger on dense geometry and camera motion, with naive fusion yielding balanced representations.
-
What VGGT Knows About Overlap: Probing Geometric Foundation Models for Co-Visibility
Frozen VGGT layers contain hierarchical co-visibility signals that a <7.5M MoE head extracts to raise Co-VisiON pairwise IoU* by >25% and multiview by ~10% over prior work.
-
Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Robotics
JITOMA keeps 3D scene graph nodes dormant until a natural-language query activates only task-relevant objects, bounding active graph size and captioning latency while preserving grounding accuracy.
-
LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation
Distillation from frontier VLMs plus E-RLVR regularization produces a 4B local model that achieves 34.5% SR on OVON while cutting inference latency by 82.8%.
-
Enabling Extensible Embodied Capabilities with Tools
Introduces Embodied Tool Protocol and tool externalization to improve embodied AI performance on perception and cognition tasks, with measured gains but limits on execution capabilities.
-
RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots
RGB-only active 3D scene graph generation unifies perception and planning to achieve depth-baseline parity and more than double object detection in active indoor exploration.
-
Fixed External Cameras as Common Prior Maps for Active 3D Scene Graph Generation
Fixed external cameras as Common Prior Maps boost initial object recall in 3D scene graph generation by up to 79% and improve active exploration efficiency.
-
EmbodiedLGR: Integrating Lightweight Graph Representation and Retrieval for Semantic-Spatial Memory in Robotic Agents
A hybrid semantic graph and retrieval-augmented system with parameter-efficient VLMs achieves state-of-the-art inference and querying speeds on embodied navigation tasks with competitive accuracy.
-
HSG: Hyperbolic Scene Graph
Hyperbolic Scene Graph (HSG) learns embeddings in hyperbolic space for better hierarchical structure in scene graphs, achieving graph IoU of 33.51 versus 25.37 for the best Euclidean baseline.
-
Memory Over Maps: 3D Object Localization Without Reconstruction
A map-free localization method stores posed RGB-D keyframes, retrieves and re-ranks them with a VLM, then fuses sparse depth for on-demand 3D target estimates, matching reconstruction-based performance on navigation b...
-
Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search
SCOUT matches LLM planners on open-world interactive object search by scoring 3D scene-graph nodes with lightweight models distilled from LLM relational priors, at far lower compute cost.
-
FloAff-Kitchen: Bridging Navigation and Manipulation via Canonical and Progressive Floor Affordance Learning
Canonical local floor geometry plus progressive skill adaptation predicts robot base placements that raise simulated kitchen mobile-manipulation success over prior FloAff methods.
-
ObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Exploration
ObsGraph is a hierarchical observation-centric scene graph that unifies representation, retrieval, and multi-scale exploration for embodied reasoning.
-
SCOUT: Semantic scene COverage via Uncertainty-guided Traversal
SCOUT integrates uncertainty-guided traversal planning with online probabilistic 3D scene graph construction to treat semantic completeness as an active objective.
-
PoseRefer: Pathway-Local Parameters for Semantically Grounded Reference Resolution
Decoupled pose-text fusion on MM-Conv reaches 31.9% top-1 accuracy and shows a learned gate changing policy based on category access in text, serving as a diagnostic against category-representation artifacts.
-
Mono-Hydra++: Real-Time Monocular Scene Graph Construction with Multi-Task Learning for 3D Indoor Mapping
Mono-Hydra++ is a monocular RGB-IMU pipeline that constructs hierarchical 3D scene graphs in real time while reporting lower trajectory error than some RGB-D baselines on indoor datasets.
-
M2H-MX: Multi-Task Semantic and Geometric Perception for Real-Time Monocular 3D Scene Graph Construction
A DINOv3-based multi-task depth/semantics front end cuts monocular ScanNet ATE by 60.7% and improves NYUDv2 dense prediction when plugged into an unmodified Mono-Hydra mapping pipeline.
-
IGV-RRT: Prior-Real-Time Observation Fusion for Active Object Search in Changing Environments
IGV-RRT improves object goal navigation in dynamic indoor environments by combining uncertainty-aware priors from 3D scene graphs with online VLM observations in a real-time tree planner.
-
OpenNavMap: Multi-Session Appearance-Based Topometric Mapping for Scalable Visual Navigation
OpenNavMap shows that an image graph plus on-demand 3D reconstruction can match structure-based maps for visual localization and navigation.
-
Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering
A training-free pipeline constructs a spatiotemporal knowledge graph from egocentric video, enabling low-latency, explainable embodied question answering.
-
N2M: Bridging Navigation and Manipulation by Learning Pose Preference from Rollout
N2M predicts preferable base poses for manipulation policies from ego-centric point clouds, learned from rollouts, lifting success from 3% to 54% in the PnPCounterToCab task.
-
Hierarchical Object Representation for Spatial Robot Perception: Points, Meshes, and Superquadrics
Introduces a hierarchical object representation pipeline from RGB-D data to meshes and superquadrics for reconstruction, map alignment, and collision checking in robotics.
-
Robust Graph Matching through Semantic Relationship Generation for SLAM
Semantic relations between objects and structural elements filter candidate graph matches in SLAM, cutting ambiguity and computation in symmetric indoor environments.
Discussion (0). Sign in to comment.