Pith.

Robotics · arXiv:2610.08780 v1 · submitted 6 Oct 2026 · 51 references

DepthWorld: 3D World Model for Robot Manipulation

Jai Bardhan, Josef Sivic, Vladimir Petrik

The paper says

“World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning.”

more of the abstract

All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. We introduce a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving <0.7 px reprojection error on 90% of episodes for external cameras). We then train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained Variational Autoencoder (VAE) unchanged. Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 7 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The calibration contribution is real and well-evidenced, but the headline RGB gain is confounded: RGB+D runs twice the latent tokens per step, and the paper never runs the token-matched control that would clean the attribution. This line is wrong

4 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 51 references say.

The paper says, at §4.1 and Appendix D.3 (Table 2, sentence b157)

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The headline +1.48 dB PSNR gain is confounded by model capacity. DepthWorld encodes six tiles (RGB and depth for each of three views) into a 72×80 joint latent, while the RGB-only baseline uses three RGB tiles in a 72×40 grid; this doubles the spatial token count and, with attention in the U-Net, roughly doubles per-step compute.

more of objection 1

Appendix D.3 calls the baseline a controlled comparison with the 'same set, split, architecture, and training budget,' but the training budget is never defined as steps versus FLOPs, and the two models do not receive the same compute per step at equal step count. No control replaces the three depth tiles with noise, a constant, or duplicated RGB while keeping the token grid at 72×80. Until such a control is run, the +1.48 dB gain cannot be attributed to depth supervision rather than to the additional learned capacity, and the Limitations paragraph (b153) does not acknowledge this confound.

The paper says, at Appendix C.8 and Appendix A.1 (Table 2, sentence b343)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.74):

“The depth metrics are not: each model is scored against the depth source it was trained on, so they mea- sure how faithfully a model reproduces its own supervision rather than its absolute metric accuracy. 11 Table A2:Ablation: Spatial latent tiling vs. other integration architectures, at∼70k training steps under the autoregressive rollout protocol (Appendix D.3).”

Pith says: Objection 2 of 4 · major

The metric-depth claim is only sparsely grounded and is otherwise self-referential. All DROID-3D depth, for both training and evaluation, comes from the learned S2M2 disparity network; the depth metrics in Table 2 and Appendix C are computed against the VAE round-trip of the same S2M2 depth, so they measure self-reproduction of the training signal, exactly as the authors note in b343.

more of objection 2

Independent metric grounding is provided for five in-lab ChArUco capture sets and 50 REALM scenes, but 71,100 episodes remain validated only against S2M2 itself. The conclusion that DepthWorld produces 'accurate metric depth for downstream geometric reasoning' therefore holds only under the untested assumption that S2M2 depth is metrically unbiased on the unvalidated DROID scenes. I would like to see either a larger independent depth-validation set or a clearer separation between the calibrated-grounding claim and the S2M2-reproduction claim in the abstract and Section 5.

The paper says, at §5 and Appendix D.4 (Table A5)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.83):

“Second, DepthWorld predicts depth in the latent space of the SVD V AE, whose encode–decode round trip alone has an error floor that PointWorld’s point representation does not have; how we account for this is described below.”

Pith says: Objection 3 of 4 · major

The comparison with PointWorld is qualified in the text but the table presentation makes it easy to over-read. In Table A5, PointWorld is scored against raw ground-truth depth while DepthWorld is scored against VAE-round-tripped ground truth, and Appendix D.4 notes that the VAE round trip alone displaces the ground truth by about 45 mm on moving points.

more of objection 3

This 'own-ceiling' protocol is disclosed and argued carefully, but the headline numbers (e.g., whole-scene Chamfer 8→9 mm for DepthWorld vs 8→18 mm for PointWorld) mix two different targets. Since the off-diagonal comparisons are said to reverse the winner, the summary in Section 5 should state explicitly that the comparative ranking is conditional on each model being scored above its own representational floor, and the main text should not present Table A5 as a direct accuracy comparison.

The paper says, at §5, Table 2

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

Several reported differences between RGB+D and RGB+D PM-DPT are very small (24.09 vs 24.11 dB PSNR; external AbsRel 0.0765 vs 0.0782), and no variance across seeds or confidence intervals is reported anywhere. Given that the PM-DPT contribution is advertised as a refinement, single-run differences of this size are not interpretable. Reporting at least two seeds or bootstrap intervals for the main Table 2 metrics would make the qualitative claims actionable.

5 minor objections

The paper says, at §1, Introduction

Pith says: minor objection 1 of 5

There is a duplicated word in 'kinematically decoupled from from the robot’s own coordinate frame'; please remove the second 'from.'

The paper says, at Figure 1 caption

Pith says: minor objection 2 of 5

The caption says DepthWorld 'capitalizes on the depth inputs,' but the inference protocol described in Appendix C.7 initializes the history buffer from the first frame and the model is not given ground-truth depth at rollout. Please clarify whether depth is an input at inference or only a training target, and adjust the caption wording accordingly.

show 3 more

The paper says, at Table 2 caption and §5

Pith says: minor objection 3 of 5

The phrase 'specially on wrist metrics' should be 'especially on wrist metrics.' Also, the depth-quality columns in Table 2 are evaluated against the VAE round-trip of the S2M2 depth; this caveat appears in Appendix C.8 but should be repeated in the Table 2 caption or Section 5 protocol for readers who do not read the appendix.

The paper says, at Appendix D.4 and D.5

Pith says: minor objection 4 of 5

The text uses unresolved placeholder citations for CoTracker3, DBSCAN, and CogVideoX-5b-I2V ('[?]'). These need to be replaced with proper references before publication.

The paper says, at Appendix C.2

Pith says: minor objection 5 of 5

The statement 'Spatial tiling performs better: it improves every RGB metric on both views' refers to Table A2, but that table compares joint RGB-depth architectures and does not include a token-matched RGB-only baseline; please add a sentence making this limitation explicit in the appendix as well as in the main text.

What would settle it

Retrain an RGB-only control on the 72×80 grid, filling the three depth-tile slots with (i) per-step i.i.d. Gaussian noise, (ii) a constant gray image, and (iii) a duplicate of the same view's RGB tile, keeping step count, batch size, and optimizer identical to RGB+D. Report external-view PSNR on the same 256 held-out trajectories and FLOPs/step for each variant. If any no-depth control moves PSNR materially from 22.63 toward 24.09, the headline is a capacity/token-count effect; only if all remain near 22.63 can 'depth supervision improves RGB' be accepted.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 7 Oct 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic disagrees 2 min 15 s · 30,350 tokens read, 10,116 written
  3. Editor wrote the verdict above 22 s · 31,130 tokens read, 3,594 written
  4. Referee major revision: 4 major objections, 5 minor 58 s · 30,640 tokens read, 4,291 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The real product here is the calibration pipeline, and it's good. Pooling all episodes of one physical robot into a single factor graph to separate shared kinematic parameters (hand-eye, joint offsets) from per-scene extrinsics is a genuine scaling insight: the Schur-complement reduction to a 13x13 global solve is clean, and the results back it up, 4.9 mm / 0.39 deg against ChArUco ground truth versus 72 mm / 3.0 deg for the PointWorld-style per-scene baseline. The calibration table (23x EE, 13x WE, 8x RD, +37 Mask IoU) is convincing, and the own-ceiling protocol in the PointWorld comparison is honest, careful evaluation work. DROID-3D, if released, is a real asset for the subfield.

The headline RGB claim is where I part company with the framing. The stress-test note is right: RGB-only runs three latent tiles per timestep, RGB+D runs six. Same nominal budget is not the same compute — attention scales with token count — and the paper never defines 'budget' as steps or FLOPs, and never runs the control that would settle it: an RGB-only model with the depth tiles replaced by noise, a constant, or duplicated RGB. The +1.48 dB is a real measured difference, but attributing it to depth supervision rather than extra capacity is untested, and the Limitations section does not mention the confound. One nuance in the paper's favor: the depth-source ablation (Table A3) shows bad depth actively hurts RGB (-2 dB), so the depth signal is doing something. But the S2M2-vs-ZED-NEURAL row, same token count, near-identical RGB at 24.07 vs 24.26 dB, is equally consistent with the gain being mostly tokens. Needs a matched-token control before I'd trust the causal story.

Other soft spots, in proportion. No code or data shipped, and DROID-3D itself unreleased, so nothing is independently reproducible. All depth metrics are scored against the S2M2 depth used in training — disclosed in C.8 — but the 'accurate metric depth at 71,000 episodes' claim rests on only five ChArUco sets and fifty REALM scenes of external grounding. The unresolved [?] citations for CoTracker3, CogVideoX-5b-I2V, and DBSCAN in Appendices D.4-D.5 break replication of the baseline protocols. The dual-branch comparison is admitted to be not 1:1.

Verdict: send to a serious referee, yes. The calibration contribution stands on its own even if the RGB-attribution story collapses. I would ask for a token-matched RGB-only control and code or data before accepting.

The strongest objection

The central empirical claim (b157) is the +1.48 dB PSNR gain of RGB+D over an 'identical' RGB-only baseline, but the comparison is not confound-free. Section 4.1 describes RGB+D as a joint latent of 72×80 made of six tiles (RGB and depth for each of three views); the RGB-only baseline has only three RGB tiles, so the latent grid is half as wide in tokens and the U-Net's per-step compute (notably attention) is correspondingly smaller. Appendix D.3 calls the baseline retrained 'on the same set, split, architecture, and training budget', but 'budget' is never defined as steps vs FLOPs; if it means steps, the depth model receives substantially more compute per step. No ablation equalizes token count while withholding depth information: Table A2 compares joint RGB-depth architectures (dual-branch, channel expansion, tiling), not an RGB-only model with the depth tiles replaced by noise, a constant, or duplicated RGB. The claimed attribution of the RGB gain to depth supervision therefore rests on an untested assumption. The paper's Limitations (b153) do not mention this confound.

I read the paper's center of gravity as the claim that depth supervision improves RGB prediction while yielding depth, because that is what b157 calls 'the central empirical finding' and what the architecture section is built around. The reader's weakest assumption (S2M2 self-reproduction) is real and is explicitly acknowledged at b343 and Appendix C.8; it constrains the absolute-depth claim but does not touch the RGB gain. The token-count confound is different: it directly threatens the controlled RGB comparison and is not acknowledged anywhere. The paper has genuine independent support—ChArUco and REALM calibration validation, a retrained equal-split baseline, and a plausible architectural story via Marigold—so I would not reject the paper. But the headline needs one additional control before the attribution is proven. I therefore keep the verdict CONDITIONAL, with the added condition that the authors supply a token-matched, no-depth baseline. My concern is not about novelty or style; it is about whether the single controlled experiment in the paper actually isolates the variable it names.

The referee's report

If the claims hold, the paper is a substantial practical contribution: it upgrades a widely used real-robot manipulation dataset with dense metric depth and robot-grounded multi-view calibration, and it shows a way to inject 3D supervision into a pretrained video-diffusion world model without destructive architectural modification. The ChArUco/REALM validation of the calibration pipeline (4.9 mm / 0.39 deg median extrinsics error) is a strong piece of evidence, and the paper is unusually candid about caveats such as the VAE round-trip evaluation floor and the held-out-trajectory (not cross-embodiment) scope. The main risk is that the headline RGB improvement is not yet attributable to depth supervision because the RGB+D model also has twice the latent token count of the RGB-only baseline; this needs a token-matched control before the central claim can be accepted at face value.

Pith's reading of the paper (a machine's summary, not the authors')

The paper tries to establish that a video world model for robot manipulation can produce rollouts that compose into a single coherent 3D scene, not just visually plausible frames. To get there it builds two things: DROID-3D, a recalibration of the DROID dataset that supplies dense metric depth and robot-grounded camera poses for over 70,000 episodes, and DepthWorld, a video-diffusion world model that predicts multi-view RGB and depth together. The central empirical claim is that adding depth as a joint prediction target improves RGB prediction itself by +1.48 dB PSNR at equal training budget, so geometric supervision is not a cost to appearance prediction but a benefit. If correct, this opens a route to world models whose rollouts can be read off for policy evaluation, improvement, and planning.

What carries the argument. Spatial latent tiling is the architectural device that carries the argument: each view's RGB and depth are encoded as separate images by the untouched SVD VAE and placed side-by-side in a 72x80 latent grid, so the single pretrained U-Net denoises both modalities jointly with only the spatial position embeddings extended. This is what allows depth supervision to be added without destructive re-initialization of the pretrained video backbone. The companion mechanism is the joint factor graph, an SE(3)-parameterized robust loss over all episodes of a robot whose Schur-complement solve reduces to a 13x13 global problem per iteration, cleanly separating per-scene camera corrections from shared kinematic parameters. A DPT point-map head initialized from VGGT lifts the predicted RGB-depth latents into robot base coordinates and supervises cross-view agreement directly in 3D.

Core claim. The core claim is that 3D-consistent world-model rollouts are attainable without sacrificing the pretrained video prior. On the data side, the paper shows that a multi-view stereo teleoperation collection with a known robot kinematic model can be lifted into supervision-quality 3D: S2M2 learned stereo disparity supplies per-view metric depth, and a joint factor graph pooling all episodes of one physical robot separates per-scene extrinsic noise from shared kinematic errors (hand-eye correction and joint offset), grounding the rig in the robot's base frame. On the modeling side, DepthWorld encodes each view's RGB and depth independently with the unmodified SVD variational autoencoder and tiles them side-by-side in a wider latent grid, so a single pretrained U-Net denoises both modalities together. The paper reports that this joint target beats an identical RGB-only baseline by +1.48 dB PSNR on external views and +1.02 dB on wrist views while producing metric depth, with a lightweight DPT point-map head supervised in the robot base frame further sharpening wrist-view depth.

The premise it rests on. Every depth number in the paper — the calibration targets that define DROID-3D and the ground truth that all DepthWorld depth accuracy is scored against — comes from the learned S2M2 stereo disparity, which is checked against real metric ground truth on only five in-lab ChArUco capture sets and 50 REALM scenes out of 71,100 total.

If the paper is right (Pith's inferences, not the authors')

World models for manipulation can be trained so their rollouts carry internally consistent geometry, making occlusion and contact reasoning possible where RGB-only models fail.

Supervising a second modality need not trade off against the first: depth supervision improved the RGB signal, so geometric training targets can act as a free gain.

The calibration approach transfers to any multi-view stereo teleoperation corpus with a known URDF, so other large manipulation datasets could be lifted into 3D without new capture hardware.

Because tiling exploits the VAE's latent capacity rather than re-initializing weights, the pattern could transfer to other pretrained video diffusion backbones.

The recalibrated extrinsics have a measurable downstream effect: point-map supervision trained against DROID-3D poses rather than factory poses improves wrist-view depth accuracy (AbsRel 0.233 to 0.220).

The paper reports that the calibration is far more robust than per-scene refinement, with no analogous per-scene failure mode to the baseline's roughly 18% non-convergence.

The +1.48 dB RGB gain is scored against real RGB pixels, so it would survive even if S2M2 depth were metrically biased; it requires only that depth is an information-bearing auxiliary signal. A cleaner control than the paper runs would corrupt or permute the depth targets and check the gain disappears.

If tiling works for depth, it likely generalizes to any modality the pretrained VAE can encode — normals, segmentation, optical flow — giving a generic recipe for joint-modality world models.

The tracked-point and Chamfer comparisons with PointWorld hint that depth-image world models drift less on static scenes than point-cloud world models; whether that holds over longer horizons and harder scenes is left open.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. We introduce a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving <0.7 px reprojection error on 90% of episodes for external cameras). We then train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained Variational Autoencoder (VAE) unchanged. Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-07

The authors' main claim, as the classifier picked it out 1.00

“Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.16 · human participants 0.05 · controlled comparison 0.94 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.93 · mentions code or data availability 0.05

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. background 1.00 0.05 0.04 asserted 0.99
All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. gap 0.93 0.11 0.06 asserted 1.00
Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors. gap 0.90 0.07 0.07 asserted 0.98
We introduce a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. method 0.89 0.96 0.05 asserted 1.00
Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset providing dense metric depth and recalibrated multi-view extrinsics (achieving <0.7 px reprojection error on 90% of episodes for external cameras). result 0.69 0.96 0.08 asserted 1.00
We then train DepthWorld, a Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, leaving the pretrained Variational Autoencoder (VAE) unchanged. method 0.79 0.95 0.12 asserted 1.00
Depth supervision improves RGB prediction itself by +1.48 dB PSNR over an identical RGB-only baseline at equal training budget, while simultaneously yielding accurate metric depth for downstream geometric reasoning. result 1.00 0.95 0.50 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-07: 1 request, 5,821 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (4)
Figure 1 of the paper
Figure 1. DepthWorld: The RGB-only baseline fails to maintain the shape of the cutlery, and the object disappears. DepthWorld capitalizes on the depth inputs to accurately maintain the cutlery’s shape throughout the rollout while producing high-quality metric depth maps. The natural fix is to supervise geometry directly during training, the recipe behind recent geometric foundation models [6, 7, 8, 9, 10, 11]. However, this approach does not directly apply to robot world models for two reasons. First, geometric foundation models earn their generalization by training on datasets that already…
Figure 2 of the paper
Figure 2. DROID-3D calibration pipeline. Given raw stereo from two external cameras and a wrist-mounted camera together with the robot’s URDF and joint encoder readings, Stage 1 recovers per-view metric depth from each stereo pair via a learned global-matching network and uses dense correspondences to enforce multi-view consistency. Stage 2 renders the URDF into each external view to ground the rig to the robot frame, then forms a joint factor graph that pools all episodes of a robot to recover per-scene extrinsic corrections (δTs ext1, δTs ext2) alongside shared kinematic parame￾ters…
Figure 3 of the paper
Figure 3. DepthWorld Architecture. Multi-view RGB and depth are independently encoded into a spatially tiled latent grid (72 × 80) for joint U-Net denoising, preserving pretrained SVD priors. Predicted future latents (x0,f ) pass through a DPT head to predict a 3D point map (X, Y, Z) and confidence logit in the robot base frame, supervised by a robust geometric loss. expansion or dual-branch networks) destroys the strong appearance and motion priors that make the model useful in the first place. Second, supervising depth independently per-view does not directly encourage 3D consistency across…
Figure 4 of the paper
Figure 4. Qualitative Result: We show two rollouts by the baseline RGB model and DepthWorld,

Figures are the authors', reproduced from the paper.

References (51)
  1. [1] Y . Guo, L. X. Shi, J. Chen, and C. Finn. Ctrl-world: A controllable generative world model for robot manipulation.arXiv preprint arXiv:2510.10125, 2025 on PithPith's reviewarXiv
  2. [2] J. Bardhan, P. Drozdik, J. Sivic, and V . Petrik. Persistent robot world models: Stabilizing multi-step rollouts via reinforcement learning.arXiv preprint arXiv:2603.25685, 2026 on PithPith's reviewarXiv
  3. [3] Y . Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation.Advances in neural information processing systems, 36:9156–9172, 2023 on Pith
  4. [4] H. Zhen, Q. Sun, H. Zhang, J. Li, S. Zhou, Y . Du, and C. Gan. Tesseract: learning 4d embodied world models.arXiv preprint arXiv:2504.20995, 2025 on PithPith's reviewarXiv
  5. [5] L. Russell, A. Hu, L. Bertoni, G. Fedoseev, J. Shotton, E. Arani, and G. Corrado. Gaia-2: A controllable multi-view generative world model for autonomous driving.arXiv preprint arXiv:2503.20523, 2025 on PithPith's reviewarXiv
  6. [6] S. Wang, V . Leroy, Y . Cabon, B. Chidlovskii, and J. Revaud. Dust3r: Geometric 3d vision made easy. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20697–20709, 2024 on Pith
  7. [7] V . Leroy, Y . Cabon, and J. Revaud. Grounding image matching in 3d with mast3r. InEuropean conference on computer vision, pages 71–91. Springer, 2024 on Pith
  8. [8] J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny. Vggt: Visual ge- ometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5294–5306, 2025 on Pith
  9. [9] Y . Wang, J. Zhou, H. Zhu, W. Chang, Y . Zhou, Z. Li, J. Chen, J. Pang, C. Shen, and T. He. pi3: Permutation-equivariant visual geometry learning.arXiv preprint arXiv:2507.13347, 2025 on PithPith's reviewarXiv
  10. [10] H. Lin, S. Chen, J. Liew, D. Y . Chen, Z. Li, G. Shi, J. Feng, and B. Kang. Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025 on PithPith's reviewarXiv
  11. [11] N. Keetha, N. M ¨uller, J. Sch¨onberger, L. Porzi, Y . Zhang, T. Fischer, A. Knapitsch, D. Zauss, E. Weber, N. Antunes, et al. Mapanything: Universal feed-forward metric 3d reconstruction. arXiv preprint arXiv:2509.13414, 2025 on PithPith's reviewarXiv
  12. [12] C. Yeshwanth, Y .-C. Liu, M. Nießner, and A. Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 12–22, 2023 on Pith
  13. [13] G. Baruch, Z. Chen, A. Dehghan, T. Dimry, Y . Feigin, P. Fu, T. Gebauer, B. Joffe, D. Kurz, A. Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data.arXiv preprint arXiv:2111.08897, 2021. 10 on PithPith's reviewarXiv
  14. [14] J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, and D. Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10901– 10911, 2021 on Pith
  15. [15] Z. Li and N. Snavely. Megadepth: Learning single-view depth prediction from internet photos. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 2041–2050, 2018 on Pith
  16. [16] Y . Yao, Z. Luo, S. Li, J. Zhang, Y . Ren, L. Zhou, T. Fang, and L. Quan. Blendedmvs: A large- scale dataset for generalized multi-view stereo networks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1790–1799, 2020 on Pith
  17. [17] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. Turner, E. Un- dersander, W. Galuba, A. Westbury, A. X. Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.arXiv preprint arXiv:2109.08238, 2021 on PithPith's reviewarXiv
  18. [18] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset.arXiv preprint arXiv:2403.12945, 2024 on PithPith's reviewarXiv
  19. [19] F. Zhu, H. Wu, S. Guo, Y . Liu, C. Cheang, and T. Kong. Irasim: A fine-grained world model for robot manipulation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9834–9844, 2025 on Pith
  20. [20] F. Jiang, Y . Chen, K. Xu, Y . Liu, H. Wang, Z. Shen, J. Lu, S. Huang, Y . Wang, C. Xie, et al. Robowm-bench: A benchmark for evaluating world models in robotic manipulation.arXiv preprint arXiv:2604.19092, 2026 on PithPith's reviewarXiv
show 31 more
  1. [21] R. Ranftl, A. Bochkovskiy, and V . Koltun. Vision transformers for dense prediction. InPro- ceedings of the IEEE/CVF international conference on computer vision, pages 12179–12188, 2021 on Pith
  2. [22] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely. Stereo magnification: Learning view synthesis using multiplane images.arXiv preprint arXiv:1805.09817, 2018 on PithPith's reviewarXiv
  3. [23] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y . Levi, Z. En- glish, V . V oleti, A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023 on PithPith's reviewarXiv
  4. [24] B. Ke, A. Obukhov, S. Huang, N. Metzger, R. C. Daudt, and K. Schindler. Repurposing diffusion-based image generators for monocular depth estimation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9492–9502, 2024 on Pith
  5. [25] W. Hu, X. Gao, X. Li, S. Zhao, X. Cun, Y . Zhang, L. Quan, and Y . Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2005–2015, 2025 on Pith
  6. [26] C. Zhang, Z. Wu, G. Lu, Y . Tang, and Z. Wang. imowm: Taming interactive multi-modal world model for robotic manipulation.arXiv preprint arXiv:2510.09036, 2025 on PithPith's reviewarXiv
  7. [27] E. Pallotta, S. M. Azar, S. Li, O. Zatsarynna, and J. Gall. Syncvp: joint diffusion for syn- chronous multi-modal video prediction. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 13787–13797, 2025 on Pith
  8. [28] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V . My- ers, M. J. Kim, M. Du, et al. Bridgedata v2: A dataset for robot learning at scale. InConference on Robot Learning, pages 1723–1736. PMLR, 2023. 11 on Pith
  9. [29] A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022 on PithPith's reviewarXiv
  10. [30] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In2024 IEEE International Conference on Robotics and Automati... on Pith
  11. [31] H.-S. Fang, H. Fang, Z. Tang, J. Liu, C. Wang, J. Wang, H. Zhu, and C. Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot.arXiv preprint arXiv:2307.00595, 2023 on PithPith's reviewarXiv
  12. [32] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y . Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots.arXiv preprint arXiv:2406.02523, 2024 on PithPith's reviewarXiv
  13. [33] R. Gong, J. Huang, Y . Zhao, H. Geng, X. Gao, Q. Wu, W. Ai, Z. Zhou, D. Terzopoulos, S.-C. Zhu, et al. Arnold: A benchmark for language-grounded task learning with continuous states in realistic 3d scenes. InProceedings of the IEEE/CVF International Conference on Computer Visi... on Pith
  14. [34] W. Huang, Y .-W. Chao, A. Mousavian, M.-Y . Liu, D. Fox, K. Mo, and L. Fei-Fei. Point- world: Scaling 3d world models for in-the-wild robotic manipulation.arXiv preprint arXiv:2601.03782, 2026 on PitharXiv
  15. [35] J. Min, Y . Jeon, J. Kim, and M. Choi. S2m2: Scalable stereo matching model for reliable depth estimation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 26729–26739, 2025 on Pith
  16. [36] J. Edstedt, D. Nordstr ¨om, Y . Zhang, G. B¨okman, J. Astermark, V . Larsson, A. Heyden, F. Kahl, M. Wadenb¨ack, and M. Felsberg. Roma v2: Harder better faster denser feature matching.arXiv preprint arXiv:2511.15706, 2025 on PithPith's reviewarXiv
  17. [37] D. Nordstr ¨om, J. Edstedt, G. B ¨okman, J. Astermark, A. Heyden, V . Larsson, M. Wadenb¨ack, M. Felsberg, and F. Kahl. Loma: Local feature matching revisited.arXiv preprint arXiv:2604.04931, 2026 on PithPith's reviewarXiv
  18. [38] M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023 on PithPith's reviewarXiv
  19. [39] J. T. Barron. A general and adaptive robust loss function. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4331–4339, 2019 on Pith
  20. [40] D. Barath, J. Noskova, M. Ivashechkin, and J. Matas. Magsac++, a fast, reliable and accurate robust estimator. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1304–1312, 2020 on Pith
  21. [41] C. Yuan, S. Joshi, S. Zhu, H. Su, H. Zhao, and Y . Gao. Roboengine: Plug-and-play robot data augmentation with semantic robot segmentation and background generation. In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7622– on Pith
  22. [42] M. Sedlacek, P. Yefanov, G. Ponimatkin, J. Bardhan, S. Pilc, M. Fourmy, E. Kazakos, C. G. Snoek, J. Sivic, and V . Petrik. Realm: A real-to-sim validated benchmark for generalization in robotic manipulation.IEEE Robotics and Automation Letters, 2026. 12 on Pith
  23. [43] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004 on Pith
  24. [44] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018 on Pith
  25. [45] B. Wen, M. Trepte, J. Aribido, J. Kautz, O. Gallo, and S. Birchfield. Foundationstereo: Zero- shot stereo matching. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5249–5260, 2025 on Pith
  26. [46] C. Ryali, Y .-T. Hu, D. Bolya, C. Wei, H. Fan, P.-Y . Huang, V . Aggarwal, A. Chowdhury, O. Poursaeed, J. Hoffman, et al. Hiera: A hierarchical vision transformer without the bells- and-whistles. InInternational conference on machine learning, pages 29441–29454. PMLR, 2023 on Pith
  27. [47] D. G. Lowe. Object recognition from local scale-invariant features. InProceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1150–1157. Ieee, 1999 on Pith
  28. [48] D. DeTone, T. Malisiewicz, and A. Rabinovich. Superpoint: Self-supervised interest point detection and description. InProceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 224–236, 2018 on Pith
  29. [49] P. Lindenberger, P.-E. Sarlin, and M. Pollefeys. Lightglue: Local feature matching at light speed. InProceedings of the IEEE/CVF international conference on computer vision, pages 17627–17638, 2023 on Pith
  30. [50] P.-E. Sarlin, A. Unagar, M. Larsson, H. Germain, C. Toft, V . Larsson, M. Pollefeys, V . Lepetit, L. Hammarstrand, F. Kahl, et al. Back to the feature: Learning robust camera localization from pixels to pose. InProceedings of the IEEE/CVF conference on computer vision and patt... on Pith
  31. [51] A. ˇS. Mikeˇst´ıkov´a, M. Fourmy, M. Cifka, J. Sivic, and V . Petrik. Alignpose: Generalizable 6d pose estimation via multi-view feature-metric alignment. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14626–14636, 2026. 13 Supplemen... on Pith
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment