R2S-EGO: Dual-Proxy Refinement for Sparse-Capture Real-to-Sim

Jie Chen; Shuai Fang; Xin Deng; Yuchen Kang; Zhenjiang Li

REVIEW 2 major objections 4 minor 46 references

R2S-EGO: Dual-Proxy Refinement for Sparse-Capture Real-to-Sim

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read R2S-EGO claims that a six-photo real-to-sim scene refined with 36 behavior-targeted, capture-anchored synthetic ego views outperforms baselines using up to 45 real photos, raising held-out robot-view PSNR from 14.226 to 19.062 dB and…

desk verdict Honest, well-controlled systems paper with a big R2S gain; the main open question is whether VGGT really registers the generated frames at the poses the method claims. read the letter →

arxiv 2608.06827 v1 pith:63G6VZTB submitted 2026-08-07 cs.RO cs.CVcs.GR

Shuai Fang , Xin Deng , Yuchen Kang , Zhenjiang Li , Jie Chen This is my paper · ORCID

classification cs.ROcs.CVcs.GR

keywords real-to-simsparse-capturereconstructionego-viewsynthesis3DGaussiansplattingpseudo-observationrefinementbehavior-scopedviewselectionsim-to-realpolicytransfergeometryproxy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to show that a sparse real-to-sim scene can be made usable for robot training without dense multi-view capture, by generating a small number of synthetic camera views that are chosen and placed according to the robot's own behavior. The proposed method, R2S-EGO, takes six registered photos of a room, uses a simulator rollout to find the ego-camera poses a robot would actually occupy during a target behavior, renders a capture-anchored geometry proxy at those poses, and asks a video-diffusion generator to produce the missing views. Those generated views are then registered and assimilated into a 3D Gaussian splatting asset as low-weight pseudo-observations, while the real photos stay as anchors. On 48 frozen Unitree G1 ego views across three Replica scenes, the refined scene renders at 19.062 dB PSNR versus 14.226 dB for the strongest reported baseline, and policies trained in it succeed at 82.5% on a real G1 sitting task versus 10.0% for the GaussGym baseline. If these results hold, sparse capture combined with targeted synthesis could replace much denser real-image capture for building robot-ready simulation scenes.

What carries the argument

The load-bearing mechanism is the dual-proxy coupling and its selection score $s_l(t)=u_l(t)\,g_l(t)$, where $u_l(t)$ is the fraction of pixels the current visual asset cannot render at candidate ego pose $t$ and $g_l(t)$ is the fraction the geometry proxy can render there; temporal non-maximum suppression and a TopK budget turn this score into a fixed set of target views each round. The robot proxy is a simulator-derived stream of executable ego-camera poses obtained by rerunning the fixed behavior controller with resampled initial conditions, which guarantees the query pose belongs to the behavior scope. The geometry proxy is composed as $M^l_{\mathrm{scaf}} = \mathrm{Compose}(\{T^*_m(P^m_{\mathrm{prior}})\}_m, \mathrm{NKSR}(P^{\neg\mathrm{prior}}_{\mathrm{obs}} \cup P(A^l)^{\neg\mathrm{prior}}))$, i.e. SAM 3D prior surfaces aligned to real captures by a similarity transform and fused with a neural-kernel surface reconstruction of the observed and current back-end points; it supplies the structural condition rendered along the generation path and the refreshed collision surface. The assimilation rule keeps real captures at unit weight and generated pseudo-observations at weight $w_{\mathrm{syn}}=0.5$, so the real images remain coordinate and appearance anchors while the generated frames extend renderable support at behavior-scoped views.

What would settle it

Render the same 36 generated terminal frames into two refined scenes, one registered with VGGT poses and one inserted at the known query poses; if the VGGT version does not match or beat the known-pose version on the 48 frozen ego views, the reported gains depend on calibration accuracy rather than on the dual-proxy selection. A cheaper check is to measure VGGT's relative pose error on generated frames against the query poses as a function of distance from the reference capture; if the error grows past a few pixels at the farthest targets, mis-registered pseudo-observations are being baked into the very views the method claims to improve.

Watch

Extended reading notes

Core claim

On its own terms, R2S-EGO's finding is that the usefulness of generated observations in real-to-sim refinement is governed less by the generator than by query selection and grounding. The paper defines a robot proxy: by rerunning a fixed behavior controller in the existing simulator, it obtains the set of ego-camera poses the embodied robot can actually reach while performing the declared behavior. A geometry proxy, formed by aligning SAM 3D shape priors to triangulated capture fragments and completing the remainder with NKSR, supplies structure at those poses. The method scores candidate poses by the product of current visual-asset deficit and geometry availability, keeps temporal local maxima up to a fixed budget, generates ego-view videos with ViewCrafter conditioned on a real reference, the relative camera motion, and the rendered geometry, then registers the terminal frames with VGGT and adds them to the 3DGS asset at half weight. The central claim is that this targeted, capture-anchored assimilation, not the generation itself, produces the reported gains: without robot-proxy allocation the PSNR drops by 2.880 dB, and one-shot refinement without round-by-round reassessment drops by 1.554 dB.

Load-bearing premise

The gain rests on the assumption that the pose-estimation software places each generated synthetic camera view accurately enough that adding those images at the estimated positions improves exactly the robot views the method targets, even though the method refuses to use the known camera positions and never measures the software's error.

Editorial extensions

If this is right

At the measured office-2 capture-count curve, no baseline setting up to 45 real input views reaches the six-view R2S-EGO PSNR, so the method's claim is not merely about matching image density but about replacing it.
The ablation pattern implies the gain decomposes: robot-proxy allocation contributes the largest single drop (2.880 dB), SAM 3D grounding contributes 2.678 dB, and iterative round-by-round reassessment contributes 1.554 dB, all under matched generation and optimization budgets.
The refined scene keeps the simulator's robot dynamics and control stack fixed, so the method is an asset-construction layer that plugs into an existing simulator rather than a new simulator or policy learner.
Policies trained on the refined asset succeed on the real G1 in 33/40 pooled trials versus 4/40 for GaussGym, with the same direction across all five paired seeds; the paper reports this as evidence that the refined ego-view asset, not in-simulation performance alone, tracks transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct stress test of the registration assumption would compare the VGGT-calibrated pseudo-observations against the same frames inserted at the known query poses; if the gap to full R2S-EGO narrows or reverses, pose calibration would be the real bottleneck and a stronger pose estimator would immediately improve the results.
The method's components are modular; any generator that accepts a reference image, relative camera motion, and a structural condition could replace ViewCrafter, and any shape prior could replace SAM 3D, so the central claim about targeted assimilation is separable from the specific public checkpoints used.
The hardware evaluation covers only sitting; if the same behavior-scoped allocation logic transfers to walking or manipulation, the capture-count savings would generalize, but that requires redefining the behavior scope and re-running the protocol, which the paper explicitly leaves to future work.
Since the evaluation uses Replica ground-truth intrinsics and extrinsics for real captures and evaluation views, the strongest numbers may not hold in physical rooms where VGGT must also estimate the real-capture calibration; the paper notes this boundary but does not quantify it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this about R2S-EGO: it's a well-scoped, honestly reported system paper that reports a large real-to-sim improvement, and the claim is mostly believable because the authors went out of their way to control the evaluation. The new content is the coupling of two existing ideas—simulator rollouts that define which ego views are behavior-relevant, and capture-anchored geometry that conditions synthetic view generation—into a fixed-budget refinement loop for 3DGS. The paper doesn't claim novelty for the components, and the related work is honest about that.

What it does well: the fixed-before protocol (six inputs, 48 targets, qualitative views frozen before running), matched reconstruction/generation budgets, five paired seeds, a disjoint development split, and a hardware comparison that shares the collision mesh and chair assets. The ablations are sensible: dropping robot-proxy allocation costs 2.88 dB, one-shot refinement costs 1.55 dB, and removing SAM 3D grounding costs 2.68 dB. The paper repeatedly states what it doesn't claim, including that the hardware counts are descriptive. That discipline makes the 19.062 dB vs 14.226 dB and 33/40 vs 4/40 numbers more credible than the usual pipeline paper.

The soft spot that matters most is the VGGT calibration of generated frames. Every generated pseudo-observation enters the 3DGS update at a pose estimated by VGGT; the query pose is deliberately not used. In the Replica benchmark, real inputs and evaluation cameras use ground-truth calibration, so the paper never measures whether these estimated poses are accurate for video-diffusion output frames. If VGGT drifts with distance from the reference capture or reacts to generated-image statistics, the pseudo-observations land where they weren't intended, and the PSNR gain could be partly an artifact of pose noise rather than deficit-targeted refinement. The paper's own limitation section (Sec. V) mentions error propagation in generated appearance and prior-completed structure but never mentions calibration. That's a real gap, and it's the first thing I'd ask an author to address—either report pose error against ground truth in Replica or run an ablation that uses the known query poses as training poses.

Minor concerns: baselines like Vanilla 3DGS and GaussGym are single fixed runs with no variance, which makes the visual gap look larger than it may be; the hardware evaluation is small and unblinded, though the paired-seed design helps; no artifacts are released, so independent verification is harder; and the visual benchmark uses Replica renders rather than sensor images, which the authors disclose. None of these sink the paper. The central claim holds up as plausible, conditional on the calibration check.

Who's this for: anyone working on real-to-sim, sparse-view reconstruction, or sim-to-real transfer with humanoids. It deserves a real peer review. I'd send it out, and I'd tell the reviewers to focus on the VGGT calibration question and the single-run baselines. If the authors can close the calibration gap, this is a strong result.

Referee Report

2 major / 4 minor

Summary. The paper presents R2S-EGO, a method for refining a sparse-capture real-to-sim scene along robot ego views without collecting additional real images. A simulator-derived robot proxy enumerates behavior-scoped executable camera poses and scores them by current visual-asset deficit combined with geometry-proxy availability; a capture-anchored geometry proxy, built from triangulated fragments, SAM 3D priors, and NKSR, supplies structural conditioning for a video-diffusion generator. Generated terminal frames are registered by VGGT, added to a 3D Gaussian splatting asset as lower-weight pseudo-observations, and the fused geometry is reinstalled as the collision surface between refinement rounds. The authors report 19.062 dB PSNR on 48 frozen G1 ego views across three Replica scenes versus 14.226 dB for the strongest R2S baseline, and 82.5% versus 10.0% real-G1 sitting success across five paired policy-training seeds, with a matched-budget evaluation protocol and component ablations.

Significance. If the reported results hold, R2S-EGO is a meaningful step toward making sparse-capture real-to-sim useful for robot learning: it shows that a small number of real images plus targeted, behavior-scoped pseudo-observation refinement can improve frozen robot-view rendering and downstream sim-to-real policy transfer. The paper's experimental hygiene is a real strength: the six inputs, 48 targets, and qualitative views were fixed before running comparisons; development and evaluation views are disjoint; the five pipeline seeds are pre-specified and paired; iterative 3DGS conditions share a 15k-step budget; the hardware comparison shares the same collision mesh and simulator stack and varies only the visual asset; and the authors transparently label descriptive statistics and scope limits. The main risk is that the pseudo-observation registration step, which is load-bearing for the targeting claim, is never validated on the actual video-diffusion output frames.

major comments (2)

[Sec. III-C, Eq. (3); Sec. IV.A] The training pose of every generated pseudo-observation is the VGGT estimate π̂_v, and the query pose is deliberately discarded ('the query pose itself is not used as its training pose'). In the Replica benchmark, the real inputs and evaluation cameras use ground-truth calibration while only the generated observations use VGGT (Sec. IV.A). The paper therefore never measures how close π̂_v is to the intended target v, nor whether VGGT's error is biased for video-diffusion output frames. If VGGT drifts with distance from the reference capture or reacts to diffusion-specific image statistics, pseudo-observations are inserted at positions other than the scored deficit; the 19.062 dB result and the 2.880 dB allocation gap (Table III) would then depend on an unmeasured calibration property rather than on deficit-targeted refinement. Please add a Replica-side calibration experiment: compare VGGT-estimated poses of the retained generated frames with the known target poses (rotation and translation errors, ideally binned by distance from the reference), and/or rerun the full pipeline using the query pose as the training pose as a sensitivity check.
[Sec. IV.C, Table III] The 'w/o iterative scene refinement' row changes multiple properties at once: it selects all 36 targets once from the initial proxy, performs a single 15k-step update, and has no intermediate NKSR/collision refresh, controller rerollout, or reranking. The 1.554 dB gap relative to full R2S-EGO therefore conflates the benefit of repeated budgeted reallocation with the benefit of refreshed geometry/collision and rerollouts. Because this row is the main evidence for the refinement-loop contribution, please add a variant that holds one of these dimensions fixed (for example, one-shot target selection but with proxy refresh and rerollout, or multi-round updates without reallocation) so the source of the improvement is identifiable.

minor comments (4)

[Sec. IV.D, Table IV] The paired seed counts (0,0,1,2,1) versus (7,7,7,6,6) are visually decisive, but since the authors explicitly describe the pooled 4/40 and 33/40 as descriptive, a paired nonparametric test or a bootstrap over seeds would make the cross-seed claim easier to quantify.
[Sec. IV.A, hardware protocol] The protocol states that 'the operator is aware of the condition.' Given that success is defined by no intervention, no load-bearing tether, and 3 s stable support, the bias risk is limited, but a blinded or automated success assessment would further strengthen the hardware result.
[Throughout] The word 'frozen' is used both for pre-registered evaluation views and for fixed model checkpoints; distinguishing 'pre-registered' from 'frozen parameters' in the text would remove a source of ambiguity.
[Sec. IV.B, Fig. 4] The figure caption and the text already clarify that error bars are SEM over 16 target views and not seed uncertainty; consider stating in the main text why the error bars are not used to compare the R2S-EGO six-view point with baseline curves.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: held-out views are frozen and pose-disjoint from all pseudo-observation targets, so the reported gains are not forced by construction.

full rationale

R2S-EGO's derivation chain is self-contained rather than circular. The 48 frozen G1 ego views were fixed before running and are pose-disjoint from the pseudo-observation inputs: the paper states 'The 48 targets are distinct from both the inputs and all retained pseudo-observation target poses.' The query pose used for generation is deliberately not used as the training pose ('VGGT then estimates that generated frame's intrinsics and pose relative to the fixed real reference before it enters the visual-asset update; the query pose itself is not used as its training pose'), so selected target views are not inserted directly at evaluation cameras. Baselines share the same six registered inputs, matched generation/retained-frame counts, and matched 3DGS optimization budgets, and the hardware comparison holds physics, collision mesh, teacher, student architecture, and SONIC/WBC stack fixed while changing only the training visual asset. Ablations (w/o robot-proxy allocation, w/o SAM 3D grounding, w/o iterative refinement) are evaluated on the same 48 frozen views and budgets, so their differences are not regression-to-fit artifacts. The paper's own limitation statement (Sec. V) acknowledges that prior-completed geometry is a structural hypothesis and that pseudo-observation updates may propagate generated-appearance errors, and the VGGT calibration of video-diffusion frames is not validated with pose-error statistics; these are robustness/correctness gaps, not circular reductions. No equation in the paper redefines the target metric as a fitted parameter, and no load-bearing claim rests on a self-citation chain.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The two proxies are algorithmic constructs (procedures that compute poses and render structural conditions), not new physical entities, forces, particles, or conserved quantities, so the invented-entities ledger is empty. The load-bearing assumptions are the fidelity of Replica GT, the completeness of the rollout-based query domain, the faithfulness of ViewCrafter frames, the accuracy of VGGT registration, and the structural correctness of SAM 3D priors; of these, VGGT pose accuracy and prior geometry correctness are the least supported by evidence in the paper.

free parameters (6)

synthetic observation weight w_syn = 0.5 (real captures 1.0)
Hand-chosen weight controlling how strongly generated pseudo-observations count in 3DGS optimization; directly shapes the refined asset (Table I, Sec. III-C).
asset alpha/coverage and geometry-validity pixel thresholds = 0.8
Thresholds defining the deficit mask u^l(t) and geometry availability g^l(t) in Eq. (2); hand-set on a disjoint development split; they determine which poses are scored as support deficits.
target-view budget K and rounds L = K=12, L=3 (at most 36 generated frames)
Fixed generation budget; selection quality and the value of iterative refinement depend on these choices (Table I).
successful rollouts per round = 12
Defines the candidate pose stream V^l_0; failed rollouts are discarded and resampled, which biases the query domain (Algorithm 1, Sec. III-B.2).
temporal NMS window = 15
Collapses near-duplicate candidate poses before TopK selection (Table I, Sec. III-B.2).
optimization and generation budgets = 5k 3DGS steps per round, 15k total; N_f=25 frames per clip
Fixed budgets matched across conditions for the comparison arms; the total 15k-step budget caps all iterative 3DGS optimization (Table I, Sec. IV.A).

assumptions (6)

domain assumption Replica Ptex rendering is a valid ground-truth proxy for robot-mounted camera appearance at calibrated poses, so PSNR/SSIM/LPIPS on these synthetic frames measure R2S scene quality.
The entire headline visual benchmark (48 frozen G1 ego views) rests on this; the paper justifies it by the difficulty of dense calibrated GT in physical scenes (Sec. IV.A).
domain assumption The fixed behavior controller's successful rollouts (12 per round after discarding failures) sample the full behavior-scoped executable query domain.
Sec. III-B.2 and Algorithm 1: 'failed rollouts are discarded and resampled'; regions of the scene where the current simulator fails are never queried, potentially excluding the most under-supported views.
domain assumption ViewCrafter-generated terminal frames are faithful appearance evidence at the target pose when conditioned on the geometry-proxy render and reference-relative motion.
Eq. (3) turns generated frames into training observations; no human or automatic verification of content correctness is applied before assimilation.
domain assumption VGGT's relative pose estimates for generated frames are accurate enough that assimilating them at the estimated pose improves the visual asset.
Sec. III-C: 'VGGT then estimates that generated frame's intrinsics and pose relative to the fixed real reference before it enters the visual-asset update; the query pose itself is not used as its training pose.' No validation of VGGT error is reported.
domain assumption Aligned SAM 3D prior surfaces are structurally correct in object regions and can serve as generation conditioning and as the refreshed collision surface.
Eq. (1) and Sec. V: the paper explicitly states prior-completed regions remain structural hypotheses and their metric or collision accuracy is not independently evaluated.
standard math Standard rigid-body kinematics and SE(3) pose composition (WTC,t = WTB(qt) BTC and relative poses pi-bar = (WTC,i)^-1 WTC,v,t).
Sec. III-A and III-C; standard forward kinematics and pose composition, no novel mathematical assumptions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of R2S-EGO: Dual-Proxy Refinement for Sparse-Capture Real-to-Sim." pith.science (2026). https://pith.science/paper/63G6VZTB

@misc{pith2026260806827,
  author       = {Pith},
  title        = {Pith review of: R2S-EGO: Dual-Proxy Refinement for Sparse-Capture Real-to-Sim},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/63G6VZTB}},
  note         = {Machine review of arXiv:2608.06827}
}

read the original abstract

Real-to-sim (R2S) depends on scene representations that render observations along robot ego trajectories, yet dense multi-view capture limits per-environment real-image capture-count efficiency, and sparse human capture can leave behavior-scoped robot views under-supported. Camera-controlled synthesis can fill missing views, but its use in R2S requires behavior-admissible queries and capture-anchored structural conditioning. We present R2S-EGO, which couples a simulator-derived robot proxy that represents the behavior-scoped executable query domain with a capture-anchored geometry proxy that supplies scene-specific structural conditions. Within this domain, fixed- budget selection targets current support deficits for which geometry support is available. The generated observations are assimilated as pseudo-observations to refine the visual asset, while real captures remain anchors. The fused geometry proxy also supplies the scene collision surface, which is refreshed between rounds. Together, these updates refine the existing simulation scene while its robot dynamics and control stack stay fixed. Across 48 frozen Unitree G1 ego views in three Replica scenes, six-view R2S-EGO reaches 19.062 dB PSNR, compared with 14.226 dB for the strongest reported R2S baseline. Across five paired policy-training seeds, R2S-EGO achieves 82.5% +/- 6.8% real-G1 sitting success, compared with 10.0% +/- 10.5% for GaussGym.

Figures

Figures reproduced from arXiv: 2608.06827 by the authors.

**Figure 1.** Behavior-aligned third- and first-person sitting rollouts in simulation and on the real Unitree G1 humanoid. Teal panels show the policy rollout in the R2S-EGO simulation scene; navy panels show the corresponding real-robot rollout, aligned at approach, reorientation, and sitting. The center columns compare the simulated ego view with the onboard head-camera view at each phase, showing closely matched appearance and… view at source ↗

**Figure 2.** R2S-EGO pipeline. R2S-EGO refines an existing simulation scene along behavior-scoped robot views. A simulator-derived robot proxy represents executable camera queries and identifies which remain under-supported, while a capture-anchored geometry proxy supplies view-aligned structural conditions. Registered ego observations update the visual asset as lower-weight pseudo-observations while real captures remain anchors… view at source ↗

**Figure 3.** Same-camera qualitative comparison on frozen G1 ego views. Each row uses the identical mounted-camera pose for GT, Vanilla 3DGS, GaussGym, Difix3D+, and R2S-EGO. The displayed routine-route view is fixed within each scene and is not chosen per method. Red boxes mark the same image regions in every column: the table rim, the office chair, and the sofa boundary, where Vanilla 3DGS and GaussGym degrade into floaters an… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Measured real-image capture-count efficiency on G1 ego views. Vanilla 3DGS, GaussGym, and Difix3D+ use nested input-image sets at the five displayed budgets; R2S-EGO remains a six-view point. Error bars are SEM over 16 fixed office 2 target views, not seed uncertainty.…

**Figure 5.** Figure 5: Two-view RGB–geometry-proxy correspondence. Each card groups one room 0 view (frames 392 and 658): the RGB observation followed by its geometry-proxy rendering. Shared mask colors denote object-level associations; the joint legend contains 21 object masks plus the back…

**Figure 6.** Figure 6: Real-G1 sitting validation. The R2S-EGO scene’s smaller ego-view gap (a) coincides with 33/40 versus 4/40 sitting success across five independently trained policies per visual asset (b). (a) At a matched approach pose, the real head-camera observation (a1) versus the s…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 38 canonical work pages

[1]

Vid2Sim: Realistic and interactive simulation from video for urban navigation,

Z. Xie, Z. Liu, Z. Peng, W. Wu, and B. Zhou, “Vid2Sim: Realistic and interactive simulation from video for urban navigation,” inProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 1581–1591

work page 2025
[2]

GaussGym: An open-source real-to-sim framework for learning locomotion from pixels,

A. Escontrela, J. Kerr, A. Allshire, J. Frey, R. Duan, C. Sferrazza, and P. Abbeel, “GaussGym: An open-source real-to-sim framework for learning locomotion from pixels,” 2025. [Online]. Available: https://arxiv.org/abs/2510.15352

arXiv 2025
[3]

RoboGSim: A real2sim2real robotic Gaussian splatting simulator,

X. Li, J. Li, Z. Zhang, R. Zhang, F. Jia, T. Wang, H. Fan, K.-K. Tseng, and R. Wang, “RoboGSim: A real2sim2real robotic Gaussian splatting simulator,” 2024. [Online]. Available: https://arxiv.org/abs/2411.11839

arXiv 2024
[4]

NeRF: Representing scenes as neural radiance fields for view synthesis,

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoor- thi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” inEuropean Conference on Computer Vision (ECCV), 2020

work page 2020
[5]

3D Gaussian splatting for real-time radiance field rendering,

B. Kerbl, G. Kopanas, T. Leimk ¨uhler, and G. Drettakis, “3D Gaussian splatting for real-time radiance field rendering,”ACM Transactions on Graphics (Proc. SIGGRAPH), vol. 42, no. 4, 2023

work page 2023
[6]

SplatSim: Zero-shot sim2real transfer of RGB manipulation policies using Gaussian splatting,

M. N. Qureshi, S. Garg, F. Yandun, D. Held, G. Kantor, and A. Silwal, “SplatSim: Zero-shot sim2real transfer of RGB manipulation policies using Gaussian splatting,” in2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 6502–6509

work page 2025
[7]

CamI2V: Camera-controlled image-to-video diffusion model,

G. Zheng, T. Li, R. Jiang, Y . Lu, T. Wu, and X. Li, “CamI2V: Camera-controlled image-to-video diffusion model,” 2024. [Online]. Available: https://arxiv.org/abs/2410.15957

arXiv 2024
[8]

CameraCtrl: Enabling camera control for video diffusion models,

H. He, Y . Xu, Y . Guo, G. Wetzstein, B. Dai, H. Li, and C. Yang, “CameraCtrl: Enabling camera control for video diffusion models,” in International Conference on Learning Representations (ICLR), 2025. [Online]. Available: https://proceedings.iclr.cc/paper files/paper/2025/ hash/f98fd73d59d8494489ea970747b91fe4-Abstract-Conference.html

work page 2025

Show all 46 references

[9]

MotionCtrl: A unified and flexible motion controller for video generation,

Z. Wang, Z. Yuan, X. Wang, Y . Li, T. Chen, M. Xia, P. Luo, and Y . Shan, “MotionCtrl: A unified and flexible motion controller for video generation,” inACM SIGGRAPH 2024 Conference Papers, 2024, pp. 1–11

2024
[10]

ViewCrafter: Taming video diffusion models for high-fidelity novel view synthesis,

W. Yu, J. Xing, L. Yuan, W. Hu, X. Li, Z. Huang, X. Gao, T.-T. Wong, Y . Shan, and Y . Tian, “ViewCrafter: Taming video diffusion models for high-fidelity novel view synthesis,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

2025
[11]

Neural kernel surface reconstruction,

J. Huang, Z. Gojcic, M. Atzmon, O. Litany, S. Fidler, and F. Williams, “Neural kernel surface reconstruction,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2023, pp. 4369–4379

2023
[12]

Reconciling reality through simulation: A real- to-sim-to-real approach for robust manipulation,

M. Torne Villasevil, A. Simeonov, Z. Li, A. Chan, T. Chen, A. Gupta, and P. Agrawal, “Reconciling reality through simulation: A real- to-sim-to-real approach for robust manipulation,” inProceedings of Robotics: Science and Systems (RSS), 2024

2024
[13]

URDFormer: A pipeline for constructing articulated simulation environments from real-world images,

Q. Chen, A. Walsman, M. Memmel, K. Mo, A. Fang, D. Fox, and A. Gupta, “URDFormer: A pipeline for constructing articulated simulation environments from real-world images,” inProceedings of Robotics: Science and Systems (RSS), 2024

2024
[14]

Wanderland: Geometrically grounded simulation for open-world embodied AI,

X. Liu, J. Li, Y . Deng, R. Chen, Y . Zhang, Y . Ma, L. Guo, Y . Li, J. Zhang, and C. Feng, “Wanderland: Geometrically grounded simulation for open-world embodied AI,” 2026. [Online]. Available: https://arxiv.org/abs/2511.20620

2026
[15]

RobotArena ∞: Scalable robot benchmarking via real-to-sim translation,

Y . Jangir, Y . Zhang, K. Yamazaki, C. Zhang, K.-H. Tu, T.-W. Ke, L. Ke, Y . Bisk, and K. Fragkiadaki, “RobotArena ∞: Scalable robot benchmarking via real-to-sim translation,” inInternational Conference on Learning Representations (ICLR), 2026. [Online]. Available: https://icl...

2026
[16]

Learning interactive real-world sim- ulators,

S. Yang, Y . Du, S. K. S. Ghasemipour, J. Tompson, L. Kaelbling, D. Schuurmans, and P. Abbeel, “Learning interactive real-world sim- ulators,” inInternational Conference on Learning Representations (ICLR), 2024, outstanding Paper Award

2024
[17]

IRASim: A fine-grained world model for robot manipulation,

F. Zhu, H. Wu, S. Guo, Y . Liu, C. Cheang, and T. Kong, “IRASim: A fine-grained world model for robot manipulation,” 2024. [Online]. Available: https://arxiv.org/abs/2406.14540

2024 arXiv
[18]

Learning visual parkour from generated images,

A. Yu, G. Yang, R. Choi, Y . Ravan, J. Leonard, and P. Isola, “Learning visual parkour from generated images,” inProceedings of the Conference on Robot Learning (CoRL), ser. Proceedings of Machine Learning Research, vol. 270, 2025, pp. 2500–2516. [Online]. Available: https://p...

2025
[19]

View-invariant policy learning via zero- shot novel view synthesis,

S. Tian, B. Wulfe, K. Sargent, K. Liu, S. Zakharov, V . C. Guizilini, and J. Wu, “View-invariant policy learning via zero- shot novel view synthesis,” inProceedings of the Conference on Robot Learning (CoRL), ser. Proceedings of Machine Learning Research, vol. 270, 2025, pp. 1...

2025
[20]

GRAIL: Generating humanoid loco-manipulation from 3D assets and video priors,

T. Xie, H. Zhang, J. Park, Z. Wang, B. Wen, J. Li, X. Li, Q. Ben, H. Weng, Y . Ye, D. Minor, T. Wang, C. Jiang, S. Fidler, J. Kautz, L. Fan, Y . Zhu, Z. Luo, U. Iqbal, and Y . Yuan, “GRAIL: Generating humanoid loco-manipulation from 3D assets and video priors,” 2026. [Online]....

2026 arXiv
[21]

pixelNeRF: Neural radiance fields from one or few images,

A. Yu, V . Ye, M. Tancik, and A. Kanazawa, “pixelNeRF: Neural radiance fields from one or few images,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021

2021
[22]

MVSplat: Efficient 3D Gaussian splatting from sparse multi-view images,

Y . Chen, H. Xu, C. Zheng, B. Zhuang, M. Pollefeys, A. Geiger, T.-J. Cham, and J. Cai, “MVSplat: Efficient 3D Gaussian splatting from sparse multi-view images,” inEuropean Conference on Computer Vision (ECCV), 2024. [Online]. Available: https: //eccv.ecva.net/virtual/2024/poster/1231

2024
[23]

DUSt3R: Geometric 3D vision made easy,

S. Wang, V . Leroy, Y . Cabon, B. Chidlovskii, and J. Revaud, “DUSt3R: Geometric 3D vision made easy,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

2024
[24]

Grounding image matching in 3D with MASt3R,

V . Leroy, Y . Cabon, and J. Revaud, “Grounding image matching in 3D with MASt3R,” inEuropean Conference on Computer Vision (ECCV), 2024

2024
[25]

3DFIRES: Few image 3D reconstruction for scenes with hidden surfaces,

L. Jin, N. Kulkarni, and D. F. Fouhey, “3DFIRES: Few image 3D reconstruction for scenes with hidden surfaces,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 9742–9751

2024
[26]

WorldMirror: Universal 3D world reconstruction with any-prior prompting,

Y . Liu, Z. Min, Z. Wang, J. Wu, T. Wang, Y . Yuan, Y . Luo, and C. Guo, “WorldMirror: Universal 3D world reconstruction with any-prior prompting,” 2025. [Online]. Available: https://arxiv.org/abs/ 2510.10726

2025 arXiv
[27]

SAM 3D: 3Dfy anything in images,

X. Chen, F.-J. Chu, P. Gleize, K. J. Liang, A. Sax, H. Tang, W. Wang, M. Guo, T. Hardin, X. Li, A. Lin, J.-W. Liu, Z. Ma, A. Sagar, B. Song, X. Wang, J. Yang, B. Zhang, P. Doll ´ar, G. Gkioxari, M. Feiszli, and J. Malik, “SAM 3D: 3Dfy anything in images,” inProceedings of the ...

2026
[28]

Stable video diffusion: Scaling latent video diffusion models to large datasets,

A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y . Levi, Z. English, V . V oleti, A. Letts, V . Jampani, and R. Rombach, “Stable video diffusion: Scaling latent video diffusion models to large datasets,” 2023

2023
[29]

DynamiCrafter: Animating open-domain images with video diffusion priors,

J. Xing, M. Xia, Y . Zhang, H. Chen, W. Yu, H. Liu, G. Liu, X. Wang, Y . Shan, and T.-T. Wong, “DynamiCrafter: Animating open-domain images with video diffusion priors,” inEuropean Conference on Computer Vision (ECCV), 2024

2024
[30]

Diffusion-generated pseudo-observations for high-quality sparse- view reconstruction,

X. Liu, J. Chen, S.-H. Kao, Y .-W. Tai, and C.-K. Tang, “Diffusion-generated pseudo-observations for high-quality sparse- view reconstruction,” inEuropean Conference on Computer Vision (ECCV), 2024. [Online]. Available: https://eccv.ecva.net/virtual/2024/ poster/2393

2024
[31]

Generative sparse-view Gaussian splatting,

H. Kong, X. Yang, and X. Wang, “Generative sparse-view Gaussian splatting,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 26 745–26 755

2025
[32]

Taming video dif- fusion prior with scene-grounding guidance for 3D Gaussian splatting from sparse inputs,

Y . Zhong, Z. Li, D. Z. Chen, L. Hong, and D. Xu, “Taming video dif- fusion prior with scene-grounding guidance for 3D Gaussian splatting from sparse inputs,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 6133– 6143

2025
[33]

RI3D: Few-shot Gaussian splatting with repair and inpainting diffu- sion priors,

A. Paliwal, X. Zhou, W. Ye, J. Xiong, R. Ranjan, and N. K. Kalantari, “RI3D: Few-shot Gaussian splatting with repair and inpainting diffu- sion priors,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 25 094–25 103

2025
[34]

ExploreGS: Explorable 3D scene reconstruction with virtual camera samplings and diffusion priors,

M. Kim, S. Jeon, I. Cho, M. Yoo, and S. J. Kim, “ExploreGS: Explorable 3D scene reconstruction with virtual camera samplings and diffusion priors,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 27 042–27 051

2025
[35]

GenFusion: Closing the loop between reconstruction and generation via videos,

S. Wu, C. Xu, B. Huang, A. Geiger, and A. Chen, “GenFusion: Closing the loop between reconstruction and generation via videos,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 6078–6088

2025
[36]

Novel view synthesis from a few glimpses via test-time natural video completion,

Y . Xu, Y . Wang, and S. X. Yu, “Novel view synthesis from a few glimpses via test-time natural video completion,” inAdvances in Neu- ral Information Processing Systems (NeurIPS), vol. 38, 2025. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2025/hash/ 4...

2025
[37]

Pointmap- conditioned diffusion for consistent novel view synthesis,

T.-A.-Q. Nguyen, L. Caraffa, J.-P. Tarel, and R. Br ´emond, “Pointmap- conditioned diffusion for consistent novel view synthesis,” inProceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2026, pp. 6360–6370

2026
[38]

A solution to the next best view problem for automated surface acquisition,

R. Pito, “A solution to the next best view problem for automated surface acquisition,”IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 21, no. 10, pp. 1016–1030, 1999

1999
[39]

Information gain-based exploration using Rao-Blackwellized particle filters,

C. Stachniss, G. Grisetti, and W. Burgard, “Information gain-based exploration using Rao-Blackwellized particle filters,” inRobotics: Science and Systems (RSS), 2005

2005
[40]

ActiveNeRF: Learning where to see with uncertainty estimation,

X. Pan, Z. Lai, S. Song, and G. Huang, “ActiveNeRF: Learning where to see with uncertainty estimation,” inEuropean Conference on Computer Vision (ECCV), 2022

2022
[41]

Ac- tiveGAMER: Active Gaussian mapping through efficient rendering,

L. Chen, H. Zhan, K. Chen, X. Xu, Q. Yan, C. Cai, and Y . Xu, “Ac- tiveGAMER: Active Gaussian mapping through efficient rendering,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 16 486–16 497

2025
[42]

AREA3D: Active reconstruction agent with unified feed-forward 3D perception and vision-language guidance,

T. Xu, S. Gan, L. Gu, Y . Li, F. Zhan, and H. Pfister, “AREA3D: Active reconstruction agent with unified feed-forward 3D perception and vision-language guidance,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 37 133–37 142

2026
[43]

MAGICIAN: Efficient long-term planning with imagined Gaussians for active mapping,

S. Li, A. Gu ´edon, S. Chen, and V . Lepetit, “MAGICIAN: Efficient long-term planning with imagined Gaussians for active mapping,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 21 606–21 615

2026
[44]

SAM 3: Segment anything with concepts,

N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris Coll-Vinent, C. Ryali, K. V . Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. R ¨adle, T. Afouras, E. Mavroudi, K. Xu, T.-H. Wu, Y . Zhou, L. Momeni, R. Hazra, ...
[45]

VGGT: Visual geometry grounded transformer,

J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny, “VGGT: Visual geometry grounded transformer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

2025
[2025]

Available: https://arxiv.org/abs/2511.16719

[Online]. Available: https://arxiv.org/abs/2511.16719

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Vid2Sim: Realistic and interactive simulation from video for urban navigation,

Z. Xie, Z. Liu, Z. Peng, W. Wu, and B. Zhou, “Vid2Sim: Realistic and interactive simulation from video for urban navigation,” inProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 1581–1591

work page 2025

[2] [2]

GaussGym: An open-source real-to-sim framework for learning locomotion from pixels,

A. Escontrela, J. Kerr, A. Allshire, J. Frey, R. Duan, C. Sferrazza, and P. Abbeel, “GaussGym: An open-source real-to-sim framework for learning locomotion from pixels,” 2025. [Online]. Available: https://arxiv.org/abs/2510.15352

arXiv 2025

[3] [3]

RoboGSim: A real2sim2real robotic Gaussian splatting simulator,

X. Li, J. Li, Z. Zhang, R. Zhang, F. Jia, T. Wang, H. Fan, K.-K. Tseng, and R. Wang, “RoboGSim: A real2sim2real robotic Gaussian splatting simulator,” 2024. [Online]. Available: https://arxiv.org/abs/2411.11839

arXiv 2024

[4] [4]

NeRF: Representing scenes as neural radiance fields for view synthesis,

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoor- thi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” inEuropean Conference on Computer Vision (ECCV), 2020

work page 2020

[5] [5]

3D Gaussian splatting for real-time radiance field rendering,

B. Kerbl, G. Kopanas, T. Leimk ¨uhler, and G. Drettakis, “3D Gaussian splatting for real-time radiance field rendering,”ACM Transactions on Graphics (Proc. SIGGRAPH), vol. 42, no. 4, 2023

work page 2023

[6] [6]

SplatSim: Zero-shot sim2real transfer of RGB manipulation policies using Gaussian splatting,

M. N. Qureshi, S. Garg, F. Yandun, D. Held, G. Kantor, and A. Silwal, “SplatSim: Zero-shot sim2real transfer of RGB manipulation policies using Gaussian splatting,” in2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 6502–6509

work page 2025

[7] [7]

CamI2V: Camera-controlled image-to-video diffusion model,

G. Zheng, T. Li, R. Jiang, Y . Lu, T. Wu, and X. Li, “CamI2V: Camera-controlled image-to-video diffusion model,” 2024. [Online]. Available: https://arxiv.org/abs/2410.15957

arXiv 2024

[8] [8]

CameraCtrl: Enabling camera control for video diffusion models,

H. He, Y . Xu, Y . Guo, G. Wetzstein, B. Dai, H. Li, and C. Yang, “CameraCtrl: Enabling camera control for video diffusion models,” in International Conference on Learning Representations (ICLR), 2025. [Online]. Available: https://proceedings.iclr.cc/paper files/paper/2025/ hash/f98fd73d59d8494489ea970747b91fe4-Abstract-Conference.html

work page 2025