Pith. sign in

REVIEW 3 major objections 5 minor 293 references

Robust 3D understanding is essential for embodied agents; inductive biases are learnable; text bridges perception to reasoning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

The thesis demonstrates that combining implicit 3D scene representations with LLM-based reasoning, using text as an interface, yields strong performance on robotic perception and spatial language tasks.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A solid PhD thesis compiling six strong papers, but the headline 'practical path to embodied spatial intelligence' is a research agenda, not a demonstrated result. the 3 major comments →

arxiv 2509.00465 v1 pith:63KKXPH5 submitted 2025-08-30 cs.RO cs.AIcs.CV

Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning

Jiading Fang This is my paper
classification cs.RO cs.AIcs.CV
keywords embodied spatial intelligenceimplicit neural representations3D scene understandinglarge language modelsspatial reasoningreferring expression resolutioncamera self-calibrationlong-horizon planning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This thesis defines embodied spatial intelligence as the ability to perceive a 3D world and act on it through natural language, decomposing it into a perception step that builds a 3D representation and an action step that reasons over it. It claims that the geometric inductive biases traditionally hard-coded into vision systems are largely learnable from data, and that implicit neural representations offer a robust, generalizable, and scalable way to build those 3D representations. On the reasoning side, it claims that large language models, though weak at mapping and navigation on their own, can be made spatially competent by transcribing 3D scenes into text, executing code for quantitative checks, and maintaining an explicit state summary during long tasks. If these claims hold, robots could recalibrate their cameras from raw video, transfer depth models across domains without retraining, fuse independently built maps, and follow complex language instructions in cluttered real environments.

Core claim

Robust 3D understanding is essential for embodied agents; many of the inductive biases needed for that understanding are learnable at scale; and a practical path toward embodied spatial intelligence lies in strategically integrating high-quality 3D structure with large-scale 2D foundation models, using text as a unifying medium. The thesis supports this with six results: sub-pixel self-calibration from raw video, a generalist depth network that transfers zero-shot, NeRF registration and blending, a benchmark showing LLM mapping gaps, 3D referring-expression resolution via transcribed scenes, and dual-LLM state-maintained planning.

What carries the argument

The thesis decomposes the action-perception mapping into V (perception from observations to a 3D representation) and F (from that representation to actions). It relies on implicit scene representations—continuous depth fields, neural radiance fields, and modular NeRF sub-maps—and on unified camera models with closed-form differentiable projection. For reasoning, it uses text as the unifying medium: detected objects are transcribed into structured descriptions, an LLM iterates with a Python interpreter for spatial calculations, and a second LLM maintains a concise state summary for long-horizon planning.

Load-bearing premise

The load-bearing premise is that transcribing 3D scenes into text preserves enough geometric information for language-guided spatial tasks; the thesis itself concedes this transcription is lossy and may discard metric distances, occlusions, or fine orientations.

What would settle it

Take Transcrib3D to referring expressions that require exact metric distances or occlusion ordering, such as 'the cup at least 30 centimeters from the wall, behind the tall bottle,' and see if accuracy drops to chance while humans succeed; if so, the claim that transcribed text retains the geometry needed for spatial reasoning is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Robots can recover from camera calibration drift on the fly using raw video and a self-supervised depth-and-pose objective.
  • Generalist transformer architectures with 3D data augmentation can replace hand-engineered geometric modules for depth estimation and still transfer across domains.
  • Large-scale environments can be represented as independently trained NeRFs, enabling distributed and privacy-preserving map building.
  • Language models show systematic gaps in mapping and navigation; providing explicit map information improves their downstream performance.
  • Transcribing 3D scenes into text with iterative code execution gives state-of-the-art 3D referring-expression resolution.
  • Maintaining an explicit world state via a second LLM improves long-horizon language-guided task execution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If 3D inductive biases are genuinely learnable at scale, specialized modules like cost volumes and epipolar losses may become optional; a direct test is to scale a generalist transformer on more diverse 3D data and see if the gap closes.
  • The text-as-unifying-medium strategy has a measurable ceiling: construct referring expressions that require metric precision, occlusion reasoning, or fine orientation; if accuracy drops to chance, the claim that transcribed text retains geometry is falsified.
  • The dual-LLM state-maintenance idea likely extends to belief maintenance in partially observable environments, storing uncertainty and hypotheses rather than just past actions.
  • The MANGO result suggests that explicit memory or map modules, not just larger models, may be the missing piece for spatial reasoning in language models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript is a PhD thesis that defines Embodied Spatial Intelligence and decomposes the robot control mapping as M = F ∘ V (Eq. 1.2). Chapter 2 develops implicit 3D scene representations: self-supervised camera self-calibration, the DeFiNe depth-field network, and NeRFuser for NeRF registration and blending. Chapter 3 develops LLM-based spatial reasoning: the MANGO text-navigation benchmark, Transcrib3D for 3D referring-expression resolution via scene transcription, and Statler for state-maintaining long-horizon planning. The thesis-level claims in Section 1.3 are that (a) robust 3D understanding is essential, (b) many inductive biases are learnable at scale, and (c) a practical path lies in integrating high-quality 3D structure with large-scale 2D foundation models. Each chapter is a previously published work evaluated on external benchmarks, with ablations for key design choices.

Significance. If the thesis-level synthesis were established, this would be a useful integrative statement of a research program: the per-chapter contributions are individually strong, with concrete benchmark results, ablations (Table 2.7, Figures 2.20–2.22), and public code for NeRFuser and Transcrib3D, as well as a hosted benchmark for MANGO. The works have already received peer review in their original venues. However, the manuscript's own central claim is not demonstrated: the scene-representation and reasoning components are never composed. The thesis is honest about the lossy nature of 3D-to-text transcription, but it does not analyze the consequences of that loss. The result is a credible research agenda rather than a validated architectural thesis.

major comments (3)
  1. [Section 1.3, Chapters 2–3] The thesis-level claim (c) is never tested end-to-end. Chapter 2 modules are evaluated on self-calibration, depth estimation, and NeRF registration/blending benchmarks; Chapter 3 modules are evaluated on text-based navigation, 3D referring-expression, and pick-and-place benchmarks. No experiment feeds a Chapter 2 representation into a Chapter 3 reasoner. In particular, Transcrib3D (§3.2) consumes Mask3D detections and an LLM code interpreter, not DeFiNe or NeRFuser, and Statler (§3.3) uses an LLM state summary without Chapter 2 representations. Thus 'a practical path... strategically integrating high-quality 3D structure with large-scale 2D foundation models' is an assertion, not a demonstrated result. Either add a concrete integration experiment or explicitly recast (c) as a research agenda.
  2. [Chapter 3 introduction (p. 64)] The author acknowledges that 'transcribing 3D scenes into language is a form of lossy compression, potentially discarding geometric details crucial for certain tasks.' This caveat directly affects the Transcrib3D mechanism, which uses text as the unifying medium for 3D referring expression resolution and downstream pick-and-place (§3.2.5). The manuscript does not quantify which geometric details are lost or when they matter for the benchmark results in Table 3.2. A concrete fidelity analysis is needed: compare transcript-based reasoning against a variant with structured numeric 3D data, and characterize failure cases as semantic vs. geometric. Without this, the 'practical path' claim is vulnerable exactly where the author identifies the risk.
  3. [Section 3.1] MANGO is presented as evidence about LLMs' navigation and mapping abilities, and the thesis uses this to motivate the broader embodied-spatial-intelligence claim. However, the benchmark is text-only and the mazes are 2D projections of text-game graphs (Figure 3.1), not 3D environments. The results therefore support a narrower claim about textual graph navigation than about 3D spatial reasoning. The thesis should explicitly delimit MANGO's scope so that the per-chapter claims are not read as supporting the 3D-specific central claim more than they do.
minor comments (5)
  1. [Eq. (2.4b)] The definition of r² reads 'm2_x + m2_x'; presumably this should be m_x² + m_y².
  2. [§3.1.4.1] The sentence 'we used the same walkthrough prefixes and questions as GPT-3.5 for a fair comparison' is repeated verbatim in consecutive paragraphs.
  3. [Figure 3.6 caption] Typo: 'necessary to to follow' should be 'necessary to follow'.
  4. [Figure 3.16 caption] Typo: 'However, the the baseline Code-as-Policies' should read 'However, the baseline Code-as-Policies'.
  5. [Prompt 3.5] Typo: 'Dome Roomm' should be 'Dome Room'.

Circularity Check

0 steps flagged

No circularity found: each chapter's results are measured against external benchmarks or held-out data, and the untested central synthesis is a missing-support issue, not a circular derivation.

full rationale

I walked the claimed derivation chain chapter by chapter. Section 2.1 learns camera intrinsics by minimizing a photometric view-synthesis loss and then evaluates the resulting parameters against AprilTag/Basalt calibration and LiDAR-based depth (EuRoC, KITTI); the evaluation targets are external to the training objective, so the sub-pixel reprojection claim is not a fitted input renamed as a prediction. Section 2.2 (DeFiNe) is trained on ScanNet RGB-D and evaluated on held-out ScanNet, 7-Scenes, and virtual-camera projections; the novel-view ground truth is generated from sensor depth and available point clouds, not from DeFiNe's own output, so the zero-shot transfer and query-vs-projection comparisons are genuine generalization tests. Section 2.3 (NeRFuser) registers and blends pretrained NeRFs and checks against ground-truth RGB-D-fused point clouds and held-out imagery; the registration and blending metrics are not constructed from the method's own predictions. The LLM chapters (MANGO, Transcrib3D, Statler) are evaluated on external benchmarks (Jericho games, ReferIt3D, ScanRefer) and simulator/real-robot rollouts, and the LLM responses are not used to define the ground-truth labels. The thesis is compiled from the author's own prior papers, so self-citation is pervasive, but no load-bearing argument reduces to a self-citation chain: the cited works are independently benchmarked, and the strongest external supports (e.g., VGGT for learnable inductive biases) are not the author's own. The main weaknesses are evidentiary, not circular: Section 1.3's claim that a practical path lies in integrating 3D structure with 2D foundation models is never tested as an end-to-end composed system, and the Section 3 preface explicitly concedes that 'transcribing 3D scenes into language is a form of lossy compression, potentially discarding geometric details crucial for certain tasks.' These are limitations of support and correctness risk, not cases where an equation or prediction reduces by construction to its own input. Accordingly, no circular step is identified.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The thesis introduces a conceptual framing but no new physical entities. Its central claims rest on standard ML assumptions and several hand-chosen hyperparameters. The free parameters are typical for deep learning systems and are ablated in part, but they do contribute to the final benchmark numbers.

free parameters (5)
  • DeFiNe loss weights lambda_s, lambda_v = tuned (ablated in Table 2.7)
    Balance depth and view synthesis losses; chosen by hand, sensitivity shown in ablation.
  • DeFiNe augmentation noise scales sigma_v, sigma_t, sigma_r = e.g., sigma_v=0.25m used in Figure 2.11b
    Control the strength of 3D augmentations; tuned on validation.
  • NeRFuser blending rate gamma = 10 (Mission Bay)
    Governs IDW blending sharpness; ablated in Figure 2.22.
  • NeRFuser proximity threshold tau = 1.2
    Filters distant NeRFs in blending; chosen by hand.
  • NeRFuser distant accumulation distance d = 0.3
    Threshold for pose filtering; chosen by hand.
axioms (5)
  • domain assumption The Unified Camera Model family (UCM/EUCM/DS) can represent the cameras considered (perspective, fisheye, catadioptric).
    Relies on Usenko et al. [1]; not derived in thesis, but needed for self-calibration validity.
  • domain assumption Input NeRFs are trained from poses that have been preprocessed in a standardized way (centered, rotated, up-aligned, scaled to [-1,1]).
    Stated in Section 2.3.3.1; necessary for pose sampling in NeRF registration. If not met, registration fails.
  • domain assumption LLMs can reason reliably over text transcripts of 3D scenes when given code execution tools and principles.
    Core to Transcrib3D and Statler; empirically tested but not guaranteed across all scenes.
  • domain assumption Mask3D object detector provides sufficient recall of objects relevant to referring expressions.
    Transcrib3D depends on the detector's output; if objects are missed, the referent may be absent.
  • standard math Standard backpropagation and Transformer architectures are taken as given.
    Used throughout; standard ML machinery.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning." pith.science (2026). https://pith.science/paper/63KKXPH5

@misc{pith2026250900465,
  author       = {Pith},
  title        = {Pith review of: Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/63KKXPH5}},
  note         = {Machine review of arXiv:2509.00465}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This thesis introduces "Embodied Spatial Intelligence" to address the challenge of creating robots that can perceive and act in the real world based on natural language instructions. To bridge the gap between Large Language Models (LLMs) and physical embodiment, we present contributions on two fronts: scene representation and spatial reasoning. For perception, we develop robust, scalable, and accurate scene representations using implicit neural models, with contributions in self-supervised camera calibration, high-fidelity depth field generation, and large-scale reconstruction. For spatial reasoning, we enhance the spatial capabilities of LLMs by introducing a novel navigation benchmark, a method for grounding language in 3D, and a state-feedback mechanism to improve long-horizon decision-making. This work lays a foundation for robots that can robustly perceive their surroundings and intelligently act upon complex, language-based commands.

Figures

Figures reproduced from arXiv: 2509.00465 by Jiading Fang.

Figure 2.1
Figure 2.1. Figure 2.1: Our self-supervised self-calibration procedure can recover accurate parameters for a wide range of cameras using a structure-from-motion objective on raw videos (EuRoC dataset, top), enabling on-the-fly re-calibration and robustness to intrinsics perturbation (bottom). Camera calibration is integral to robotics and computer vision algorithms that seek to infer geometric properties of the scene from visua… view at source ↗
Figure 2.2
Figure 2.2. Figure 2.2: Our self-supervised self-calibration architecture. We use gradients from the photometric loss to update the parameters of a unified camera model ( [PITH_FULL_IMAGE:figures/full_fig_p031_2_2.png] view at source ↗
Figure 2.3
Figure 2.3. Figure 2.3: The Unified Camera Model [1] used in our self-calibration pipeline. Points are projected onto a unit sphere before being projected onto an image plane of a standard pinhole camera offset by α 1−α from the sphere center. aforementioned view synthesis objective alone. At the start of self-supervised depth and pose training, rather than pre-calibrating the camera parameters, we initialize the camera with “d… view at source ↗
Figure 2.4
Figure 2.4. Figure 2.4: EuRoC rectification results using images from the calibration sequences. Each column visualizes the results rendered using (left) the Basalt calibrated intrinsics and (right) our learned intrinsics. The top row shows that detected (small circles) and reprojected (big circles) corners are close using both calibration methods. The bottom row shows the same images after rectification. toolbox and compute ca… view at source ↗
Figure 2
Figure 2. Figure 2: , we see that our learned parameters allow for the rectified grid to [PITH_FULL_IMAGE:figures/full_fig_p035_2.png] view at source ↗
Figure 2.5
Figure 2.5. Figure 2.5: EuRoC perturbation test, showing how our proposed learning￾based method is able to recover from changes in camera parameters for online self-calibration. a camera based on a potentially incorrect prior calibration. Generally, this requires the capture of new calibration data. Instead, we can initialize our parameter vectors with this initial calibration (in this setting, a perturbation of Basalt calibrat… view at source ↗
Figure 2.6
Figure 2.6. Figure 2.6: Self-supervised monocular pointcloud for EuRoC, obtained by unprojecting predicted depth with our learned camera parameters (input image on the bottom right). lenging dataset that involves cluttered indoor sequences with six-DoF motion. Compared to the per-frame distorted camera models of Gordon et al. [4] and Vasiljevic et al. [44], we achieve significantly better absolute relative error, especially wit… view at source ↗
Figure 2.7
Figure 2.7. Figure 2.7: Qualitative depth estimation results on non-pinhole datasets with (a) fisheye and (b) catadioptric images. 2.1.5 Conclusion We proposed a procedure to self-calibrate a family of general camera models using self-supervised depth and pose estimation as a proxy task. We rigorously evaluated the quality of the resulting camera models, demonstrating sub-pixel calibration accuracy comparable to manual target-b… view at source ↗
Figure 2.8
Figure 2.8. Figure 2.8: Our Depth Field Networks (DeFiNe) achieve state of the art in multi-view depth estimation, while also enabling predictions from arbitrary viewpoints. Modern 3D computer vision leverages learning to boost geometric reasoning, mapping image data to classical structures such as cost volumes or epipolar constraints to improve matching. These architectures are specialized according to the particular problem, … view at source ↗
Figure 2.9
Figure 2.9. Figure 2.9: Overview of our proposed DeFiNe architecture, and the em￾beddings used to encode and decode information for depth and view synthesis. 2.2.3.1 Perceiver IO Perceiver IO [85] is a recent extension of the Perceiver [89] architecture. The Perceiver architecture alleviates one of the main weaknesses of Transformer￾based methods, namely the quadratic scaling of self-attention with input size. This is achieved … view at source ↗
Figure 2.10
Figure 2.10. Figure 2.10: Geometric augmentations. (a) Information from camera i is projected onto a virtual camera at Tv, creating additional supervision from other viewpoints. (b) Noise T ′ 0 is introduced to the canonical camera at T0, and then propagated to other cameras to preserve relative scene geometry. consistent geometric latent scene representation. Therefore, in this section we propose a series of 3D augmentations to… view at source ↗
Figure 2.11
Figure 2.11. Figure 2.11: Depth estimation results on ScanNet-Stereo. (a) We outperform contemporary methods by a large margin. (b) Depth estimation results on virtual cameras using different values for σv at training and test time. use a custom split to select 2000 samples from 90 scenes not covered in the training set. Each training sample includes a target frame and a context of [−3, 3] frames with stride 3. Each test sample … view at source ↗
Figure 2.12
Figure 2.12. Figure 2.12: Reconstructed two-view pointclouds, from ScanNet-Stereo. DeFiNe pointclouds are generated using both depth maps and RGB images queried from our learned latent representation. GT DeFiNe GT DeFiNe [PITH_FULL_IMAGE:figures/full_fig_p055_2_12.png] view at source ↗
Figure 2.13
Figure 2.13. Figure 2.13: Depth estimation and view synthesis results on ScanNet. Although view synthesis is not our primary goal, it can be achieved with minimal modifications, and we show that it improves depth estimation performance. of traditional relative viewing rays; ( [PITH_FULL_IMAGE:figures/full_fig_p055_2_13.png] view at source ↗
Figure 2.14
Figure 2.14. Figure 2.14: Depth estimation results from novel viewpoints. frames onto these new viewpoints. We evaluate both standard depth estimation networks [15, 68, 96] as well as DeFiNe itself, that can be used to either explicitly project information from encoded frames onto new viewpoints (projection), or query from the latent representation at that same location (query) [PITH_FULL_IMAGE:figures/full_fig_p058_2_14.png] view at source ↗
Figure 2
Figure 2. Figure 2: reports results in terms of root mean squared error (RMSE) [PITH_FULL_IMAGE:figures/full_fig_p058_2.png] view at source ↗
Figure 2.15
Figure 2.15. Figure 2.15: Overall framework of NeRFuser We present NeRFuser , a novel framework that extends the representational capacity of neural radiance fields (NeRFs) to produce high-fidelity representa￾tions of large-scale scenes. Integral to our approach is the decomposition of spatially extended environments into a collection of small-scale scenes, each represented by an individual NeRF model (i.e., a sub-map) produced … view at source ↗
Figure 2.16
Figure 2.16. Figure 2.16: Qualitative comparison of blending methods. Our proposed [PITH_FULL_IMAGE:figures/full_fig_p062_2_16.png] view at source ↗
Figure 2.17
Figure 2.17. Figure 2.17: Illustration of NeRF renderings and their accumulations. Column [PITH_FULL_IMAGE:figures/full_fig_p067_2_17.png] view at source ↗
Figure 2.18
Figure 2.18. Figure 2.18: Illustration of IDW-based blending methods: IDW-2D, IDW-3D [PITH_FULL_IMAGE:figures/full_fig_p069_2_18.png] view at source ↗
Figure 2.19
Figure 2.19. Figure 2.19: NeRF blending with IDW-based methods on the Mission Bay [PITH_FULL_IMAGE:figures/full_fig_p071_2_19.png] view at source ↗
Figure 2
Figure 2. Figure 2: , distant accumulation-based filtering removes mostly bad images [PITH_FULL_IMAGE:figures/full_fig_p073_2.png] view at source ↗
Figure 2.20
Figure 2.20. Figure 2.20: Registration error and time consumption against varying [PITH_FULL_IMAGE:figures/full_fig_p074_2_20.png] view at source ↗
Figure 2.21
Figure 2.21. Figure 2.21: Effect of re-rendering poses on NeRF registration. [PITH_FULL_IMAGE:figures/full_fig_p075_2_21.png] view at source ↗
Figure 2.22
Figure 2.22. Figure 2.22: Effect of blending rate γ in IDW-based blending. Ablation of γ in IDW-based blending We study the effect of blending rate γ in IDW-based blending on Object-Centric Indoor Scenes. Specifically, we use ground-truth transformations and set distance test ratio τ = 1.8. We geometrically sample γ in [10−2 , 103 ]. For each sampled γ, we blend NeRFs with all IDW-based methods and report the results averaged ov… view at source ↗
Figure 3.1
Figure 3.1. Figure 3.1: Map of Zork-I. Arrows denote the direction of travel during the [PITH_FULL_IMAGE:figures/full_fig_p080_3_1.png] view at source ↗
Figure 3.2
Figure 3.2. Figure 3.2: Success rates of the examined models on (a) DF and (b) RF [PITH_FULL_IMAGE:figures/full_fig_p089_3_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: presents the success rates of all models. For each kind of question [PITH_FULL_IMAGE:figures/full_fig_p089_3.png] view at source ↗
Figure 3.3
Figure 3.3. Figure 3.3: Success rates of GPT-3.5 and GPT-4 broken down into individual [PITH_FULL_IMAGE:figures/full_fig_p091_3_3.png] view at source ↗
Figure 3.3
Figure 3.3. Figure 3.3: For example, both Zenon and OMNIQuest stay at the lower-left [PITH_FULL_IMAGE:figures/full_fig_p092_3_3.png] view at source ↗
Figure 3.4
Figure 3.4. Figure 3.4: Playing minigames. corners of the hard-question plots in [PITH_FULL_IMAGE:figures/full_fig_p093_3_4.png] view at source ↗
Figure 3.5
Figure 3.5. Figure 3.5: The overall Transcrib3D framework, which takes as input the colored point-cloud and referring expression (in green), and outputs the ID or bounding box of the referent object. To resolve the referring expression “the chair in the corner of the room, between the white and yellow desks”, the framework needs to locate the pillow in the green box, while all other pillows in red boxes are distractors. If robo… view at source ↗
Figure 3.6
Figure 3.6. Figure 3.6: Transcrib3D enables a robot to resolve complex 3D referring [PITH_FULL_IMAGE:figures/full_fig_p097_3_6.png] view at source ↗
Figure 3
Figure 3. Figure 3: illustrates our proposed Transcrib3D framework. Given the input [PITH_FULL_IMAGE:figures/full_fig_p099_3.png] view at source ↗
Figure 3.7
Figure 3.7. Figure 3.7: Illustration of the iterative code generation and reasoning process. [PITH_FULL_IMAGE:figures/full_fig_p101_3_7.png] view at source ↗
Figure 3.8
Figure 3.8. Figure 3.8: Qualitative comparisons between Transcrib3D (ours, in [PITH_FULL_IMAGE:figures/full_fig_p105_3_8.png] view at source ↗
Figure 3.9
Figure 3.9. Figure 3.9: Qualitative comparison of the grounding performance of (top) CaP [PITH_FULL_IMAGE:figures/full_fig_p109_3_9.png] view at source ↗
Figure 3.10
Figure 3.10. Figure 3.10: Typical failure cases in Transcrib3D, with ground-truth bounding [PITH_FULL_IMAGE:figures/full_fig_p110_3_10.png] view at source ↗
Figure 3.11
Figure 3.11. Figure 3.11: Overall Framework of Statler There has been a significant research interest in employing large language models to empower intelligent robots with complex reasoning. Existing work focuses on harnessing their abilities to reason about the histories of their actions and observations. In this paper, we explore a new dimension in which large language models may benefit robotics planning. In particular, we pr… view at source ↗
Figure 3.12
Figure 3.12. Figure 3.12: Model accuracies on the three-cups-and-a-ball shell game. LLM+State is a simplified version of our proposed Statler framework. For each method, the solid line shows how its accuracy a(n) changes with the number of swaps n. The dashed line is the relative accuracy: r(n) = a(n)/a(1). Intuitively, it measures how fast the performance decreases from a hypothetically perfect one-swap performance. Note that L… view at source ↗
Figure 3.13
Figure 3.13. Figure 3.13: Examples of simulations that show the result of executing different [PITH_FULL_IMAGE:figures/full_fig_p117_3_13.png] view at source ↗
Figure 3.14
Figure 3.14. Figure 3.14: The simulated domains we consider include a Pick-and-Place; b [PITH_FULL_IMAGE:figures/full_fig_p119_3_14.png] view at source ↗
Figure 3.15
Figure 3.15. Figure 3.15: Examples that show the result of querying LLMs with and without [PITH_FULL_IMAGE:figures/full_fig_p120_3_15.png] view at source ↗
Figure 3.16
Figure 3.16. Figure 3.16: A comparison of the resulting behavior for (top) Code-as-Policies [PITH_FULL_IMAGE:figures/full_fig_p123_3_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

293 extracted references · 18 canonical work pages · 13 internal anchors

  1. [1]

    The double sphere camera model,

    V. Usenko, N. Demmel, and D. Cremers, “The double sphere camera model,” in Proceedings of the International Conference on 3D Vision (3DV) , 2018, pp. 552–560

    2018

  2. [2]

    Visual- inertial mapping with non-linear factor recovery,

    V. Usenko, N. Demmel, D. Schubert, J. Stueckler, and D. Cremers, “Visual- inertial mapping with non-linear factor recovery,” IEEE Robotics and Automa- tion Letters, vol. 5, no. 2, pp. 422–429, 2020

    2020

  3. [3]

    The EuRoC micro aerial vehicle datasets,

    M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik, and R. Siegwart, “The EuRoC micro aerial vehicle datasets,” International Journal of Robotics Research, vol. 35, no. 10, pp. 1157–1163, 2016

    2016

  4. [4]

    Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras,

    A. Gordon, H. Li, R. Jonschkowski, and A. Angelova, “Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  5. [5]

    PointGroup: Dual-set point grouping for 3D instance segmentation,

    L. Jiang, H. Zhao, S. Shi, S. Liu, C.-W. Fu, and J. Jia, “PointGroup: Dual-set point grouping for 3D instance segmentation,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , 2020

    2020

  6. [6]

    Mask3D: Mask transformer for 3D semantic instance segmentation,

    J. Schult, F. Engelmann, A. Hermans, O. Litany, S. Tang, and B. Leibe, “Mask3D: Mask transformer for 3D semantic instance segmentation,” in Proc. IEEE Int’l Conf. on Robotics and Automation (ICRA) , 2023

    2023

  7. [7]

    What’s “up

    A. Kamath, J. Hessel, and K.-W. Chang, “What’s “up” with vision-language models? investigating their struggle with spatial reasoning,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  8. [8]

    Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilities,

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “Spa- tialvlm: Endowing vision-language models with spatial reasoning capabilities,” 115 in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  9. [9]

    Spatialrgpt: Grounded spatial reasoning for vision-language models,

    X. Cheng, X. Wang, R. Zhao, X. Wang, L. Zhang, and S. Z. Li, “Spatialrgpt: Grounded spatial reasoning for vision-language models,” in Advances in Neural Information Processing Systems (NeurIPS) , 2024

    2024

  10. [10]

    PointNet++: Deep hierarchical feature learning on point sets in a metric space,

    C. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deep hierarchical feature learning on point sets in a metric space,” in Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  11. [11]

    KinectFusion: Real-time dense surface mapping and tracking,

    R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon, “KinectFusion: Real-time dense surface mapping and tracking,” in Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR) , 2011

    2011

  12. [12]

    Octomap: An efficient probabilistic 3d mapping framework based on octrees,

    A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard, “Octomap: An efficient probabilistic 3d mapping framework based on octrees,” Autonomous Robots, vol. 34, no. 3, pp. 189–206, 2013

    2013

  13. [13]

    Dtam: Dense track- ing and mapping in real-time,

    R. A. Newcombe, S. J. Lovegrove, and A. J. Davison, “Dtam: Dense track- ing and mapping in real-time,” in 2011 IEEE International Conference on Computer Vision (ICCV) . IEEE, 2011, pp. 2320–2327

    2011

  14. [14]

    PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,

    D. Sun, X. Yang, M. Liu, and J. Kautz, “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2018, pp. 8934–8943

    2018

  15. [15]

    Digging into self- supervised monocular depth prediction,

    C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow, “Digging into self- supervised monocular depth prediction,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2019

    2019

  16. [16]

    Depth field networks for generalizable multi-view scene representation,

    V. Guizilini, I. Vasiljevic, J. Fang, R. Ambru, G. Shakhnarovich, M. R. Walter, and A. Gaidon, “Depth field networks for generalizable multi-view scene representation,” in European Conference on Computer Vision. Springer, 2022, pp. 245–262

    2022

  17. [17]

    Input- level inductive biases for 3D reconstruction,

    W. Yifan, C. Doersch, R. Arandjelovi´ c, J. Carreira, and A. Zisserman, “Input- level inductive biases for 3D reconstruction,” arXiv preprint arXiv:2112.03243 , 2021. 116

    Pith/arXiv arXiv 2021

  18. [18]

    Delira: Self-supervised depth, light, and radiance fields,

    V. Guizilini, I. Vasiljevic, J. Fang, R. Ambrus, S. Zakharov, V. Sitzmann, and A. Gaidon, “Delira: Self-supervised depth, light, and radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 17 935–17 945

    2023

  19. [19]

    Vggt: Visual geometry grounded transformer,

    J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny, “Vggt: Visual geometry grounded transformer,” in Proceedings of the Computer Vision and Pattern Recognition Conference , 2025, pp. 5294–5306

    2025

  20. [20]

    NeRF: Representing scenes as neural radiance fields for view synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” in Proceedings of the European Conference on Computer Vision (ECCV), 2020

    2020

  21. [21]

    Nerfuser: Large-scale scene repre- sentation by nerf fusion,

    J. Fang, S. Lin, I. Vasiljevic, V. Guizilini, R. Ambrus, A. Gaidon, G. Shakhnarovich, and M. R. Walter, “Nerfuser: Large-scale scene repre- sentation by nerf fusion,” 2023

    2023

  22. [22]

    Di-nerf: Distributed nerf for collabora- tive learning with relative pose refinement,

    M. Asadi, K. Zareinia, and S. Saeedi, “Di-nerf: Distributed nerf for collabora- tive learning with relative pose refinement,” IEEE Robotics and Automation Letters, 2024

    2024

  23. [23]

    Transcrib3d: 3d referring expression resolution through large language models,

    J. Fang, X. Tan, S. Lin, I. Vasiljevic, V. Guizilini, H. Mei, R. Ambrus, G. Shakhnarovich, and M. R. Walter, “Transcrib3d: 3d referring expression resolution through large language models,” arXiv preprint arXiv:2404.19221 , 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth,

    J. E. Cutting and P. M. Vishton, “Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth,” in Perception of space and motion . Elsevier, 1995, pp. 69–117

    1995

  25. [25]

    Mango: A benchmark for evaluating mapping and navigation abilities of large language models,

    P. Ding, J. Fang, P. Li, K. Wang, X. Zhou, M. Yu, J. Li, M. R. Walter, and H. Mei, “Mango: A benchmark for evaluating mapping and navigation abilities of large language models,” arXiv preprint arXiv:2403.19913 , 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Do as I can, 117 not as I say: Grounding language in robotic affordances,

    b. Ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. T. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L. Luu, K....

    2023

  27. [28]

    Self-supervised camera self-calibration from video,

    J. Fang, I. Vasiljevic, V. Guizilini, R. Ambrus, G. Shakhnarovich, A. Gaidon, and M. R. Walter, “Self-supervised camera self-calibration from video,” in 2022 International Conference on Robotics and Automation (ICRA) , 2022, pp. 8468–8475

    2022

  28. [29]

    Structure-from-motion revisited,

    J. L. Sch¨ onberger and J.-M. Frahm, “Structure-from-motion revisited,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4104–4113

    2016

  29. [30]

    Pixelwise view selection for unstructured multi-view stereo,

    J. L. Sch¨ onberger, E. Zheng, M. Pollefeys, and J.-M. Frahm, “Pixelwise view selection for unstructured multi-view stereo,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2016, pp. 501–518

    2016

  30. [31]

    Orb-slam: a versatile and accurate monocular slam system,

    R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, “Orb-slam: a versatile and accurate monocular slam system,” IEEE transactions on robotics , vol. 31, no. 5, pp. 1147–1163, 2015

    2015

  31. [32]

    NeRF: Representing scenes as neural radiance fields for view synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” in Proceedings of the European Conference on Computer Vision (ECCV), 2020, pp. 405–421

    2020

  32. [33]

    Implicit neural representations with periodic activation functions,

    V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” Advances in neural information processing systems, vol. 33, pp. 7462–7473, 2020

    2020

  33. [34]

    A flexible new technique for camera calibration,

    Z. Zhang, “A flexible new technique for camera calibration,”IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 22, no. 11, pp. 1330–1334, 2000

    2000

  34. [35]

    A flexible technique for accurate omnidirectional camera calibration and structure from motion,

    D. Scaramuzza, A. Martinelli, and R. Siegwart, “A flexible technique for accurate omnidirectional camera calibration and structure from motion,” in Proceedings of the IEEE International Conference on Computer Vision Systems (ICVS), 2006, pp. 45–45. 118

    2006

  35. [36]

    A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,

    J. Kannala and S. S. Brandt, “A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 28, no. 8, pp. 1335–1340, 2006

    2006

  36. [37]

    A general imaging model and a method for finding its parameters,

    M. D. Grossberg and S. K. Nayar, “A general imaging model and a method for finding its parameters,” in Proceedings of the International Conference on Computer Vision (ICCV) , vol. 2, 2001, pp. 108–115

    2001

  37. [38]

    Using vanishing points for camera calibration,

    B. Caprile and V. Torre, “Using vanishing points for camera calibration,” International Journal on Computer Vision , vol. 4, no. 2, pp. 127–139, 1990

    1990

  38. [39]

    A stratified approach to metric self-calibration,

    M. Pollefeys and L. Van Gool, “A stratified approach to metric self-calibration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1997, pp. 407–412

    1997

  39. [40]

    Camera calibration from vanishing points in image of architectural scenes,

    R. Cipolla, T. Drummond, and D. P. Robertson, “Camera calibration from vanishing points in image of architectural scenes,” in Proceedings of the British Machine Vision Conference (BMVC) , 1999, pp. 382–391

    1999

  40. [41]

    DeepCalib: A deep learning approach for automatic intrinsic calibration of wide field-of-view cameras,

    O. Bogdan, V. Eckstein, F. Rameau, and J.-C. Bazin, “DeepCalib: A deep learning approach for automatic intrinsic calibration of wide field-of-view cameras,” in Proceedings of the ACM SIGGRAPH European Conference on Visual Media Production, 2018

    2018

  41. [42]

    Unsupervised CNN for single view depth estimation: Geometry to the rescue,

    R. Garg, V. K. Bg, G. Carneiro, and I. Reid, “Unsupervised CNN for single view depth estimation: Geometry to the rescue,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2016, pp. 740–756

    2016

  42. [43]

    Unsupervised learning of depth and ego-motion from video,

    T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, “Unsupervised learning of depth and ego-motion from video,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 1851–1858

    2017

  43. [44]

    Neural ray surfaces for self-supervised learning of depth and ego-motion,

    I. Vasiljevic, V. Guizilini, R. Ambrus, S. Pillai, W. Burgard, G. Shakhnarovich, and A. Gaidon, “Neural ray surfaces for self-supervised learning of depth and ego-motion,” in Proceedings of the International Conference on 3D Vision (3DV), 2020

    2020

  44. [45]

    Hartley and A

    R. Hartley and A. Zisserman, Multiple view geometry in computer vision . Cambridge university press, 2003

    2003

  45. [46]

    Bradski and A

    G. Bradski and A. Kaehler, “OpenCV,” Dr. Dobb’s Journal of Software Tools , vol. 3, 2000. 119

    2000

  46. [47]

    Extending kalibr: Calibrating the extrinsics of multiple IMUs and of individual axes,

    J. Rehder, J. Nikolic, T. Schneider, T. Hinzmann, and R. Siegwart, “Extending kalibr: Calibrating the extrinsics of multiple IMUs and of individual axes,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2016, pp. 4304–4311

    2016

  47. [48]

    A perception-driven autonomous urban vehicle,

    J. Leonard, J. How, S. Teller, M. Berger, S. Campbell, G. Fiore, L. Fletcher, E. Frazzoli, A. Huang, S. Karaman, O. Koch, Y. Kuwata, D. Moore, E. Olson, S. Peters, J. Teo, R. Truax, M. Walter, D. Barrett, A. Epstein, K. Maheloni, K. Moyer, T. Jones, R. Buckley, M. Antone, R. Galejs, S. Krishnamurthy, and J. Williams, “A perception-driven autonomous urban ...

    2008

  48. [49]

    Autonomous driving in urban environments: Boss and the Urban Challenge,

    C. Urmson, J. Anhalt, D. Bagnell, C. Baker, R. Bittner, M. Clark, J. Dolan, D. Duggins, T. Galatali, C. Geyer et al. , “Autonomous driving in urban environments: Boss and the Urban Challenge,” Journal of Field Robotics , vol. 25, no. 8, pp. 425–466, 2008

    2008

  49. [50]

    Lens distortion for close-range photogrammetry,

    J. G. Fryer and D. C. Brown, “Lens distortion for close-range photogrammetry,” Photogrammetric Engineering and Remote Sensing , vol. 52, pp. 51–58, 1986

    1986

  50. [51]

    A unifying theory for central panoramic systems and practical implications,

    C. Geyer and K. Daniilidis, “A unifying theory for central panoramic systems and practical implications,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2000, pp. 445–461

    2000

  51. [52]

    An enhanced unified camera model,

    B. Khomutenko, G. Garcia, and P. Martinet, “An enhanced unified camera model,” IEEE Robotics and Automation Letters , vol. 1, no. 1, pp. 137–144, 2015

    2015

  52. [53]

    DeepFo- cal: A method for direct focal length estimation,

    S. Workman, C. Greenwell, M. Zhai, R. Baltenberger, and N. Jacobs, “DeepFo- cal: A method for direct focal length estimation,” in Proceedings of the IEEE International Conference on Image Processing (ICIP) , 2015, pp. 1369–1373

    2015

  53. [54]

    Horizon lines in the wild,

    S. Workman, M. Zhai, and N. Jacobs, “Horizon lines in the wild,” arXiv preprint arXiv:1604.02129, 2016

    Pith/arXiv arXiv 2016

  54. [55]

    Radial lens distortion correction using convolutional neural networks trained with synthesized images,

    J. Rong, S. Huang, Z. Shang, and X. Ying, “Radial lens distortion correction using convolutional neural networks trained with synthesized images,” in Proceedings of the Asian Conference on Computer Vision , 2016, pp. 35–49

    2016

  55. [56]

    FishEyeRectNet: A multi-context collaborative deep network for fisheye image rectification,

    X. Yin, X. Wang, J. Yu, M. Zhang, P. Fua, and D. Tao, “FishEyeRectNet: A multi-context collaborative deep network for fisheye image rectification,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 469–484. 120

    2018

  56. [57]

    Deep single image camera calibration with radial distortion,

    M. Lopez, R. Mari, P. Gargallo, Y. Kuang, J. Gonzalez-Jimenez, and G. Haro, “Deep single image camera calibration with radial distortion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11 817–11 825

    2019

  57. [58]

    A perceptual measure for deep single image camera calibration,

    Y. Hold-Geoffroy, K. Sunkavalli, J. Eisenmann, M. Fisher, E. Gambaretto, S. Hadap, and J.-F. Lalonde, “A perceptual measure for deep single image camera calibration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018

    2018

  58. [59]

    Single view metrology in the wild,

    R. Zhu, X. Yang, Y. Hold-Geoffroy, F. Perazzi, J. Eisenmann, K. Sunkavalli, and M. Chandraker, “Single view metrology in the wild,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2020, pp. 316–333

    2020

  59. [60]

    DeepPTZ: Deep self-calibration for PTZ cameras,

    C. Zhang, F. Rameau, J. Kim, D. M. Argaw, J.-C. Bazin, and I. S. Kweon, “DeepPTZ: Deep self-calibration for PTZ cameras,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2020, pp. 1041–1049

    2020

  60. [61]

    Learning perspective undistortion of portraits,

    Y. Zhao, Z. Huang, T. Li, W. Chen, C. LeGendre, X. Ren, A. Shapiro, and H. Li, “Learning perspective undistortion of portraits,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2019, pp. 7849–7859

    2019

  61. [62]

    Learning to recover 3D scene shape from a single image,

    W. Yin, J. Zhang, O. Wang, S. Niklaus, L. Mai, S. Chen, and C. Shen, “Learning to recover 3D scene shape from a single image,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021, pp. 204–213

    2021

  62. [63]

    Progressively complementary network for fisheye image rectification using appearance flow,

    S. Yang, C. Lin, K. Liao, C. Zhang, and Y. Zhao, “Progressively complementary network for fisheye image rectification using appearance flow,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 6348–6357

    2021

  63. [64]

    A deep ordinal distortion estimation approach for distortion rectification,

    K. Liao, C. Lin, and Y. Zhao, “A deep ordinal distortion estimation approach for distortion rectification,” IEEE Transactions on Image Processing , vol. 30, pp. 3362–3375, 2021

    2021

  64. [65]

    Deep geometry-aware camera self- calibration from video,

    A. Hagemann, M. Knorr, and C. Stiller, “Deep geometry-aware camera self- calibration from video,” in Proceedings of the IEEE/CVF International Con- ference on Computer Vision , 2023, pp. 3438–3448

    2023

  65. [66]

    Robust self-supervised extrinsic self-calibration,

    T. Kanai, I. Vasiljevic, V. Guizilini, A. Gaidon, and R. Ambrus, “Robust self-supervised extrinsic self-calibration,” in 2023 IEEE/RSJ International 121 Conference on Intelligent Robots and Systems (IROS) . IEEE, 2023, pp. 1932–1939

    2023

  66. [67]

    Digging into self- supervised monocular depth estimation,

    C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow, “Digging into self- supervised monocular depth estimation,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2019, pp. 3828–3838

    2019

  67. [68]

    3D packing for self-supervised monocular depth estimation,

    V. Guizilini, R. Ambrus, S. Pillai, A. Raventos, and A. Gaidon, “3D packing for self-supervised monocular depth estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020

    2020

  68. [69]

    SuperDepth: Self-supervised, super- resolved monocular depth estimation,

    S. Pillai, R. Ambrus, and A. Gaidon, “SuperDepth: Self-supervised, super- resolved monocular depth estimation,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , 2019

    2019

  69. [70]

    Are we ready for autonomous driv- ing? The KITTI vision benchmark suite,

    A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driv- ing? The KITTI vision benchmark suite,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2012, pp. 3354–3361

    2012

  70. [71]

    nuScenes: A multimodal dataset for au- tonomous driving,

    H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuScenes: A multimodal dataset for au- tonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020, pp. 11 621–11 631

    2020

  71. [72]

    Depth map prediction from a single image using a multi-scale deep network,

    D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” arXiv preprint arXiv:1406.2283, 2014

    Pith/arXiv arXiv 2014

  72. [73]

    Calibrating and centering quasi- central catadioptric cameras,

    M. Sch¨ onbein, T. Strauß, and A. Geiger, “Calibrating and centering quasi- central catadioptric cameras,” in Proceedings of the IEEE International Con- ference on Robotics and Automation (ICRA) , 2014, pp. 4443–4450

    2014

  73. [74]

    Automatic differentiation in PyTorch,

    A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in PyTorch,” in Advances in Neural Information Processing Systems (NeurIPS) , 2017

    2017

  74. [75]

    AprilTag: A robust and flexible visual fiducial system,

    E. Olson, “AprilTag: A robust and flexible visual fiducial system,” in Pro- ceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2011, pp. 3400–3407

    2011

  75. [76]

    End-to-end learning of geometry and context for deep stereo 122 regression,

    A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach, and A. Bry, “End-to-end learning of geometry and context for deep stereo 122 regression,” in Proceedings of the International Conference on Computer Vision (ICCV), 2017, pp. 66–75

    2017

  76. [77]

    RAFT-Stereo: Multilevel recurrent field transforms for stereo matching,

    L. Lipson, Z. Teed, and J. Deng, “RAFT-Stereo: Multilevel recurrent field transforms for stereo matching,” in Proceedings of the International Conference on 3D Vision (3DV) , 2021, pp. 218–227

    2021

  77. [78]

    RAFT: Recurrent all-pairs field transforms for optical flow,

    Z. Teed and J. Deng, “RAFT: Recurrent all-pairs field transforms for optical flow,” in Proceedings of the European Conference on Computer Vision (ECCV), 2020

    2020

  78. [79]

    DeepV2D: Video to depth with differentiable structure from motion,

    ——, “DeepV2D: Video to depth with differentiable structure from motion,” in Proceedings of the International Conference on Learning Representations (ICLR), 2020

    2020

  79. [80]

    Neural fields in visual computing and beyond,

    Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar, “Neural fields in visual computing and beyond,” arXiv preprint arXiv:2111.11426 , 2021

    Pith/arXiv arXiv 2021

  80. [81]

    pixelNeRF: Neural radiance fields from one or few images,

    A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelNeRF: Neural radiance fields from one or few images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021

    2021

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.