Pith. sign in

REVIEW 4 major objections 4 minor 66 references

SimVS: Simulating World Inconsistencies for Robust View Synthesis

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Simulating world inconsistencies trains view synthesis to handle real-world captures.

desk verdict Genuinely new idea for handling sparse inconsistent captures via video-model-simulated training data, but the main quantitative claim leans on a questionable evaluation protocol that drops the hardest timesteps for baselines. read the letter →

arxiv 2412.07696 v1 pith:2ADR26EQ submitted 2024-12-10 cs.CV cs.AIcs.GRcs.LG

Alex Trevithick , Roni Paiss , Philipp Henzler , Dor Verbin , Rundi Wu , Hadi Alzayer , Ruiqi Gao , Ben Poole
show 4 more authors
Jonathan T. Barron Aleksander Holynski Ravi Ramamoorthi Pratul P. Srinivasan
This is my paper · ORCID
classification cs.CVcs.AIcs.GRcs.LG
keywords multi-viewharmonizationvideodiffusionmodelsworldinconsistencysimulationsparseviewsynthesisrobustnoveldynamicsceneslightingvariationgenerativedataaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Casual multi-view captures are rarely consistent: people move, lights shift, and shadows change, which breaks the static-scene assumption behind most view-synthesis models. This paper proposes to simulate exactly those inconsistencies with a pretrained video diffusion model, generating, for each view of an existing consistent multi-view dataset, a video whose frames show plausible scene motion or lighting changes while the camera stays fixed. Those simulated inconsistent frames are then used to train a multi-view harmonization network that takes a small set of inconsistent photos plus one reference image and outputs a set of images consistent with the reference state. The trained network outperforms the prior state of the art and both heuristic augmentation and synthetic-rendered training data on the DyCheck dynamics benchmark (PSNR 16.73 vs. 15.59) and on a new real-world lighting-variation benchmark (20.98 vs. 18.26), enabling static 3D reconstructions from casual sparse captures.

What carries the argument

The load-bearing mechanism is the pairing of a video-model augmentation stage with a multiview diffusion harmonization stage. In the first stage, an image- and text-conditioned video diffusion model samples, for every view of a consistent multi-view capture, a video under a shared 'static shot' prompt plus a negative prompt that forbids camera movement; randomly sampled frames become the inconsistent conditioning images while the original images serve as ground-truth consistent targets. In the second stage, a latent multiview diffusion model, fine-tuned from an existing multi-view diffusion backbone, concatenates the autoencoder latents of the inconsistent images and a binary reference mask to the noisy target latents and ray maps, and is trained with the standard diffusion denoising loss to predict the consistent target latents. This two-stage design lets the model learn, from millions of simulated frames, to transfer any chosen reference state across all views.

What would settle it

Generate simulated training videos for a sample of prompts and measure the optical flow between the conditioning frame and sampled frames: if a large fraction of clips show global translation or homographic warps consistent with camera motion rather than scene-only motion, the assumed pose alignment is violated. A second end-to-end check would be to render a synthetic scene under two known lighting states with ground-truth poses, feed the three sparse views through the harmonization model, and test whether the reconstructed geometry and held-out views match the ground-truth state as closely as the paper's results suggest.

Watch

Extended reading notes

Core claim

Building a training set of paired inconsistent and consistent multi-view images is the bottleneck for robust sparse view synthesis; collecting real multi-condition captures is impractical, and heuristics like random tints or sparse flow fields do not capture genuine 3D scene changes. The paper's central claim is that a video diffusion model can generate this paired data: conditioned on an image and a text prompt describing motion or lighting change (prefix 'static shot', with a negative prompt suppressing camera motion), it produces videos whose frames are inconsistent with the input image while the camera pose remains valid. Sampling one frame per video per input view gives inconsistent observations aligned with known poses and with ground-truth consistent images from the original dataset. A multiview diffusion model is then fine-tuned to map the reference image and up to seven inconsistent posed images to consistent target images, supervised by the original captures. The result is a harmonization model that reconciles sparse, unordered, inconsistent captures into a consistent set suitable for standard dense reconstruction.

Load-bearing premise

The load-bearing premise is that the video model can alter the scene's motion or lighting while leaving the camera perfectly still, so the original camera poses and the ground-truth consistent images remain valid supervision for the simulated inconsistent frames.

Editorial extensions

If this is right

  • A single harmonization pass converts sparse inconsistent captures into consistent images, so offline dense reconstruction methods can be applied directly to casual captures.
  • Because the reference image selects the desired state, the same input set can be used to reconstruct the scene under any of the observed motion or lighting states.
  • The approach turns video diffusion models into training-data generators, so improvements in video-model fidelity should translate directly into better robust view synthesis.
  • The same augmentation recipe applies to other consistency-demanding tasks such as monocular depth estimation and pose estimation from dynamic scenes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test, but the same simulation strategy could generate paired data for other inverse problems where real paired captures are scarce, such as deblurring, reflection removal, or exposure bracketing.
  • An extension the paper mentions but does not implement is replacing the static-camera video model with a camera-controlled video model, which could synthesize multi-view-consistent videos in one sampling pass rather than two separate stages.
  • A likely scaling behavior as video models improve is that pose-estimation robustness becomes the binding constraint; the paper already notes pose estimation as a limitation.
  • Because the training inconsistencies inherit the video model's biases, an over-production of particular artifacts such as morphing shapes could be silently baked into the harmonization model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents SimVS, a method for robust novel-view synthesis from sparse and inconsistent captures. It uses a pretrained image-to-video diffusion model (Lumiere) to generate, for each view of an existing multiview dataset, a video with simulated scene motion or lighting changes while aiming to keep the camera static; random frames from these videos serve as inconsistent conditioning images paired with ground-truth consistent multiview images. A multiview diffusion model (initialized from CAT3D) is finetuned to harmonize up to seven inconsistent inputs plus a reference image into consistent views, which are then densified with CAT3D and rendered with Zip-NeRF. The authors evaluate on DyCheck for dynamics and on a newly captured lighting dataset, reporting improved PSNR/SSIM/LPIPS over CAT3D baselines and over heuristic or Objaverse-based augmentation.

Significance. SimVS is a useful demonstration that video diffusion models can serve as generative data sources for 3D vision tasks that lack paired inconsistent-consistent training data. The paper's strengths include the scale of the generated data (roughly 6M/12M frames), the comparison against heuristic augmentation and synthesized Objaverse data, and the introduction of a real captured lighting benchmark. The harmonization model's ability to produce consistent outputs from sparse inconsistent inputs is visually compelling. However, the paper does not release code or models, and the main quantitative evidence has a significant evaluation-protocol weakness; the contribution is therefore promising but not yet fully established.

major comments (4)
  1. [§5.1, Table 1] The central quantitative claim in Table 1 is computed after discarding 4 of 28 DyCheck timesteps because 'COLMAP was unable to register the test images for at least one of the baselines,' while COLMAP never fails for the proposed method. Since registration failure is itself a failure mode of the baselines on hard inconsistent captures, this selected-denominator protocol likely overstates the reported margin; with only 24 timesteps and a 1.14 dB PSNR advantage over CAT3D (all images), the word 'significantly' is not justified without error bars or significance testing. Please re-run the comparison on all 28 timesteps with a failure-inclusive protocol, for example by reporting success rates separately or assigning a low-quality score to unregistered timesteps, and report per-timestep scores with variance.
  2. [§5.1] The evaluation recomputes poses per method with COLMAP on the generated samples, so the reported metrics jointly reflect view-synthesis quality and pose-estimation success. Because SimVS produces more consistent images, its COLMAP poses may be more reliable, making it unclear how much of the PSNR gain comes from better view synthesis rather than from easier pose estimation. Please report pose accuracy separately, use fixed poses for all methods where possible, or provide a sensitivity analysis of the final metrics to pose errors.
  3. [§3.1, Eq. (2)] The validity of the paired training data rests on the assumption that Lumiere simulates scene motion or lighting changes while keeping the camera exactly static, so that the original camera poses remain valid for synthesized frames; the paper relies on the 'static shot' prefix and a negative prompt to enforce this. No quantitative verification of camera stillness is provided. Please add a check, for example by estimating ego-motion or optical flow on generated videos and reporting camera-motion magnitudes, or otherwise analyze how residual camera motion in the simulated data could affect the harmonization model and transfer to real captures.
  4. [Supplement E] The dynamics heuristic-augmentation ablation is trained for only 12k iterations because it 'quickly overfits to copying' and 12k is where the loss on the held-out OOD data is minimized, whereas the proposed model is trained for 48k; this is a form of validation-set selection and also makes the comparison unequal in compute. Please report results for the heuristic baseline at matched training length, justify why early stopping on OOD loss is fair, and state exactly which held-out split was used so that the ablation is not tuned on the evaluation set.
minor comments (4)
  1. [Tables 1 and 3] The same model reports PSNR 16.73 in Table 1 and 16.60 in Table 3 for dynamics, and Supplement Table 1 also reports 16.60; please clarify whether these numbers come from different subsets, sampling seeds, or evaluation protocols.
  2. [§5.2] The newly captured lighting dataset is central to the lighting experiments, but the paper does not state the number of scenes, the number of frames per lighting condition, or the criteria for selecting the target illumination; please provide these details and consider releasing the dataset.
  3. [§2.2 and References] The paper states that concurrent work CAT4D leverages the generative augmentation strategy proposed here; since CAT4D shares authors, please clarify the temporal relation and how the novelty of this augmentation strategy is attributed between the two papers.
  4. [Figure 8] The bottom rows are brightened for visualization, but the brightness factor is not stated; please specify the factor or include an unadjusted version in the supplement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training and evaluation pipeline is empirically grounded, and self-citations to CAT3D/Lumiere are building blocks rather than load-bearing reductions.

full rationale

The paper's claimed derivation chain—synthesize inconsistent multiview frames with a video model, train a harmonization model against the original consistent frames, then densify and reconstruct—is not circular by construction. The training loss in Eq. (4) is supervised by ground-truth consistent latents z1:7 from existing datasets, while the 'inconsistent' conditioning frames are produced by a pretrained video model; no quantity later reported as a prediction is a fitted constant of these equations. The reported gains on DyCheck and the new lighting dataset are measured on held-out real views, not recovered from the training objective. The CAT3D and Lumiere citations include overlapping authors, but both are used as ready-made backbones and pretrained models, and the central claim depends on the measured generalization of the resulting pipeline; therefore these self-citations are not load-bearing in the sense of forcing the conclusion. Some choices are tuned on the evaluation setting (e.g., switching CAT3D to 5 conditioning frames because it 'works slightly better', and discarding 4/28 DyCheck timesteps where COLMAP fails for baselines but not for SimVS); these are legitimate correctness and robustness concerns, not circularity, since the comparison is not made equal by definition. Overall, no step reduces to its own input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. It relies on several domain assumptions about the fidelity of pretrained models and external datasets. The main free parameters are hand-chosen prompts and hyperparameters that shape the simulated inconsistency distribution.

free parameters (4)
  • Lighting prompt set = 6 predefined prompts, e.g., 'a bright light casts shadows', 'the light slowly dims from bright to dark'
    Hand-chosen set of prompts to elicit large lighting variations from Lumiere (Sec F.1). These prompts determine the distribution of simulated lighting inconsistencies and are not derived from theory.
  • Video model guidance weight and sampling steps = guidance=6.0, 250 DDPM sampling steps at 128x128, then 250 at 1024x1024
    Chosen empirically for Lumiere sampling (Sec G). These hyperparameters affect the quality and diversity of simulated inconsistencies.
  • Number of conditioning frames in harmonization model = up to 7 with dropout; test uses 3 for lighting, 7 for dynamics
    The model is trained with up to 7 inconsistent conditioning images and dropout; the test-time number of conditioning frames is task-dependent (Sec 4.2, 5.1, 5.2).
  • CAT3D finetuning configuration for dynamics = 5 conditioning images predicting 3
    The authors report that conditioning on 5 images and predicting 3 'works slightly better' than the original configuration (Sec 5.1 and supplement B). This is a test-set-driven tuning choice.
assumptions (4)
  • domain assumption Lumiere video model can generate videos with significant scene changes while preserving the input camera pose and static geometry when prompted with 'static shot' and a negative prompt.
    Invoked in Sec 3.1, Eq. (2), where videos are sampled and frames are treated as inconsistent but camera-aligned with the input image. If this fails, the synthesized inconsistent frames are misaligned with the ground-truth consistent targets.
  • domain assumption Pretrained models CAT3D, Lumiere, and Gemini are available and can be used as described.
    The method and evaluation rely on these models (Sec 3.1, 4.1). Their availability and behavior are external dependencies.
  • domain assumption Mannequin Challenge and RealEstate10k provide multiview captures of static scenes that are consistent, from which inconsistent versions can be simulated.
    These datasets are used to generate training data (Sec 3.1). If the captures contain inherent inconsistencies, the training targets would be corrupted.
  • domain assumption COLMAP and Hierarchical Localization provide accurate camera poses for the test captures after harmonization.
    Evaluation relies on re-estimated poses (Sec 5.1, 5.2). The paper notes pose estimation remains a limitation for sparse inconsistent captures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SimVS: Simulating World Inconsistencies for Robust View Synthesis." pith.science (2026). https://pith.science/paper/2ADR26EQ

@misc{pith2026241207696,
  author       = {Pith},
  title        = {Pith review of: SimVS: Simulating World Inconsistencies for Robust View Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ADR26EQ}},
  note         = {Machine review of arXiv:2412.07696}
}
read the original abstract

Novel-view synthesis techniques achieve impressive results for static scenes but struggle when faced with the inconsistencies inherent to casual capture settings: varying illumination, scene motion, and other unintended effects that are difficult to model explicitly. We present an approach for leveraging generative video models to simulate the inconsistencies in the world that can occur during capture. We use this process, along with existing multi-view datasets, to create synthetic data for training a multi-view harmonization network that is able to reconcile inconsistent observations into a consistent 3D scene. We demonstrate that our world-simulation strategy significantly outperforms traditional augmentation methods in handling real-world scene variations, thereby enabling highly accurate static 3D reconstructions in the presence of a variety of challenging inconsistencies. Project page: https://alextrevithick.github.io/simvs

Figures

Figures reproduced from arXiv: 2412.07696 by the authors.

Figure 1
Figure 1. We show results of our model applied to a casual in-the-wild capture. (a) Given [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A comparison of real world state changes, those simulated [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Our method’s overall pipeline. (a) Given a dataset of multiview images [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Samples from our multiview diffusion harmonization [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Given the reference and inputs in Fig [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results for the DyCheck [14] dataset for our model, two CAT3D baselines, and two of our ablations. The depth maps are inset on the bottom left. Images are cropped for visualization. Compared to CAT3D (all images), our method generates coherent 3D scenes des…
Figure 8
Figure 8. Figure 8: Qualitative results for 3 scenes from our captured lighting dataset. For each scene, we display the renders from the learned NeRFs given the 3 input images on the left. We show two states for each scene, with the renders outlined in blue corresponding to the upper inpu…
Figure 1
Figure 1. Figure 1: Qualitative comparison to Shape of Motion [ [PITH_FULL_IMAGE:figures/full_fig_p012_1.png]
Figure 2
Figure 2. Figure 2: Our model incorporating more context given an increasing number of images. Given the (additional) inputs on the left, our model [PITH_FULL_IMAGE:figures/full_fig_p013_2.png]
Figure 3
Figure 3. Figure 3: The meta-prompt used to generate dynamics captions on the Mannequin Challenge dataset [ [PITH_FULL_IMAGE:figures/full_fig_p015_3.png]
Figure 4
Figure 4. Figure 4: We show example samples from the lighting data we sampled. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 38 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Building rome in a day

    Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in a day. Communications of the ACM, 54 0 (10): 0 105--112, 2011

    work page 2011

  3. [3]

    Understanding the impact of negative prompts: When and how do they take effect? arXiv preprint arXiv:2406.02965, 2024

    Yuanhao Ban, Ruochen Wang, Tianyi Zhou, Minhao Cheng, Boqing Gong, and Cho-Jui Hsieh. Understanding the impact of negative prompts: When and how do they take effect? arXiv preprint arXiv:2406.02965, 2024

    arXiv 2024

  4. [4]

    Lumiere: A Space-Time Diffusion Model for Video Generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, and Inbar Mosseri. Lumiere: A Space-Time Diffusion Model for Video Generation . arXiv:2401.12945, 2024

    arXiv 2024

  5. [5]

    Barron, Ben Mildenhall, Dor Verbin, Pratul P

    Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields . ICCV, 2023

    work page 2023

  6. [6]

    Optimizing the Latent Space of Generative Networks

    Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam. Optimizing the Latent Space of Generative Networks . ICML, 2018

    work page 2018

  7. [7]

    Instructpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392--18402, 2023

    2023

  8. [8]

    Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo

    Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14124--14133, 2021

    work page 2021

Show all 66 references
  1. [9]

    Swag: Splatting in the wild images with appearance-conditioned gaussians

    Hiba Dahmani, Moussab Bennehar, Nathan Piasco, Luis Roldao, and Dzmitry Tsishkou. Swag: Splatting in the wild images with appearance-conditioned gaussians. In European Conference on Computer Vision, pages 325--340. Springer, 2025

    2025

  2. [10]

    Objaverse: A Universe of Annotated 3D Objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A Universe of Annotated 3D Objects . CVPR, 2022

    2022

  3. [11]

    Tapir: Tracking any point with per-frame initialization and temporal refinement

    Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages ...

    2023

  4. [12]

    Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds

    Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang, Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, et al. Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds. arXiv preprint arXiv:2403.20309, 2, 2024

    2024 arXiv

  5. [13]

    Fast dynamic radiance fields with time-aware neural voxels

    Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nie ner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pages 1--9, 2022

    2022

  6. [14]

    Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image

    Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In European Conference on Computer Vision, pages 241--258. Springer, 2025

    2025

  7. [15]

    Monocular Dynamic View Synthesis: A Reality Check

    Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular Dynamic View Synthesis: A Reality Check . NeurIPS, 2022

    2022

  8. [16]

    Srinivasan, Jonathan T

    Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole. CAT3D: Create Anything in 3D with Multi-View Diffusion Models . NeurIPS, 2024

    2024

  9. [17]

    Lrm: Large reconstruction model for single image to 3d

    Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400, 2023

    2023 arXiv

  10. [18]

    Geonerf: Generalizing nerf with geometry priors

    Mohammad Mahdi Johari, Yann Lepoittevin, and Fran c ois Fleuret. Geonerf: Generalizing nerf with geometry priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18365--18375, 2022

    2022

  11. [19]

    Cotracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07635, 2023

    2023 arXiv

  12. [20]

    Repurposing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9492--9502, 2024

    2024

  13. [21]

    WildGaussians: 3D Gaussian Splatting in the Wild

    Jonas Kulhanek, Songyou Peng, Zuzana Kukelova, Marc Pollefeys, and Torsten Sattler. WildGaussians: 3D Gaussian Splatting in the Wild . NeurIPS, 2024

    2024

  14. [22]

    MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds

    Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Daniilidis. MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds . arXiv:2405.17421, 2024

    2024 arXiv

  15. [23]

    Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William T. Freeman. Learning the Depths of Moving People by Watching Frozen People . CVPR, 2019

    2019

  16. [24]

    Neural scene flow fields for space-time view synthesis of dynamic scenes

    Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498--6508, 2021

    2021

  17. [25]

    Dynibar: Neural dynamic image-based rendering

    Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4273--4284, 2023

    2023

  18. [26]

    Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models

    Hanwen Liang, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N Plataniotis, Yao Zhao, and Yunchao Wei. Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models. arXiv preprint arXiv:2405.16645, 2024

    2024 arXiv

  19. [27]

    One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization

    Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024 a

    2024

  20. [28]

    Zero-1-to-3: Zero-shot one image to 3d object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9298--9309, 2023 a

    2023

  21. [29]

    Sora: A review on background, technology, limitations, and opportunities of large vision models

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024 b

    2024 arXiv

  22. [30]

    Robust Dynamic Radiance Fields

    Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust Dynamic Radiance Fields . CVPR, 2023 b

    2023

  23. [31]

    Wonder3D: Single Image to 3D using Cross-Domain Diffusion

    Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3D: Single Image to 3D using Cross-Domain Diffusion . CVPR, 2024

    2024

  24. [32]

    Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections . CVPR, 2021

    2021

  25. [33]

    Neural rerendering in the wild

    Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla. Neural rerendering in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6878--6887, 2019

    2019

  26. [34]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis . ECCV, 2020

    2020

  27. [35]

    Barron, Sofien Bouaziz, Dan B Goldman, Steven M

    Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable Neural Radiance Fields . ICCV, 2021 a

    2021

  28. [36]

    Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields

    Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228, 2021 b

    2021 arXiv

  29. [37]

    D-nerf: Neural radiance fields for dynamic scenes

    Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318--10327, 2021

    2021

  30. [38]

    From coarse to fine: Robust hierarchical localization at large scale

    Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In CVPR, 2019

    2019

  31. [39]

    Superglue: Learning feature matching with graph neural networks

    Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938--4947, 2020

    2020

  32. [40]

    Structure-from-motion revisited

    Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104--4113, 2016

    2016

  33. [41]

    Abhishek Sharma, Adams Yu, Ali Razavi, Andeep Toor, Andrew Pierson, Ankush Gupta, Austin Waters, Aäron van den Oord, Daniel Tanis, Dumitru Erhan, Eric Lau, Eleni Shaw, Gabe Barth-Maron, Greg Shaw, Han Zhang, Henna Nandwani, Hernan Moraldo, Hyunjik Kim, Irina Blok, Jakob Bauer,...

    2024

  34. [42]

    MVDream: Multi-view Diffusion for 3D Generation

    Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view Diffusion for 3D Generation . arXiv:2308.16512, 2023

    2023 arXiv

  35. [43]

    Extranerf: Visibility-aware view extrapolation of neural radiance fields with diffusion models

    Meng-Li Shih, Wei-Chiu Ma, Lorenzo Boyice, Aleksander Holynski, Forrester Cole, Brian Curless, and Janne Kontkanen. Extranerf: Visibility-aware view extrapolation of neural radiance fields with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and ...

    2024

  36. [44]

    Realmdreamer: Text-driven 3d scene generation with inpainting and depth diffusion

    Jaidev Shriram, Alex Trevithick, Lingjie Liu, and Ravi Ramamoorthi. Realmdreamer: Text-driven 3d scene generation with inpainting and depth diffusion. arXiv preprint arXiv:2404.07199, 2024

    2024 arXiv

  37. [45]

    Photo tourism: exploring photo collections in 3D

    Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3D . SIGGRAPH, 2006

    2006

  38. [46]

    Dynamic gaussian marbles for novel view synthesis of casual monocular videos

    Colton Stearns, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari, Gordon Wetzstein, and Leonidas Guibas. Dynamic gaussian marbles for novel view synthesis of casual monocular videos. arXiv preprint arXiv:2406.18717, 2024

    2024 arXiv

  39. [47]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv:2312.11805, 2023

    2023 arXiv

  40. [48]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part II 16, pages 402--419. Springer, 2020

    2020

  41. [49]

    Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video

    Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollh \"o fer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International C...

    2021

  42. [50]

    Grf: Learning a general radiance field for 3d representation and rendering

    Alex Trevithick and Bo Yang. Grf: Learning a general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15182--15192, 2021

    2021

  43. [51]

    Sparf: Neural radiance fields from sparse and noisy poses

    Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4190--4200, 2023

    2023

  44. [53]

    DUSt3R: Geometric 3D Vision Made Easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3R: Geometric 3D Vision Made Easy . CVPR, 2024 b

    2024

  45. [54]

    We-gs: An in-the-wild efficient 3d gaussian representation for unconstrained photo collections

    Yuze Wang, Junyi Wang, and Yue Qi. We-gs: An in-the-wild efficient 3d gaussian representation for unconstrained photo collections. arXiv preprint arXiv:2406.02407, 2024 c

    2024 arXiv

  46. [55]

    Nerfiller: Completing scenes via generative 3d inpainting

    Ethan Weber, Aleksander Holynski, Varun Jampani, Saurabh Saxena, Noah Snavely, Abhishek Kar, and Angjoo Kanazawa. Nerfiller: Completing scenes via generative 3d inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20731--20741, 2024

    2024

  47. [56]

    Cat4d: Create anything in 4d with multi-view video diffusion models

    Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng, Jonathan T Barron, and Aleksander Holynski. Cat4d: Create anything in 4d with multi-view video diffusion models. arXiv preprint arXiv:2411.18613, 2024 a

    2024 arXiv

  48. [57]

    Reconfusion: 3d reconstruction with diffusion priors

    Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P Srinivasan, Dor Verbin, Jonathan T Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  49. [58]

    Spatialtracker: Tracking any 2d pixels in 3d space

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20406--20417, 2024

    2024

  50. [59]

    Wild-gs: Real-time novel view synthesis from unconstrained photo collections

    Jiacong Xu, Yiqun Mei, and Vishal M Patel. Wild-gs: Real-time novel view synthesis from unconstrained photo collections. arXiv preprint arXiv:2406.10373, 2024

    2024 arXiv

  51. [60]

    Depth anything v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406.09414, 2024

    2024 arXiv

  52. [61]

    pixelnerf: Neural radiance fields from one or few images

    Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4578--4587, 2021

    2021

  53. [62]

    Gaussian in the wild: 3d gaussian splatting for unconstrained image collections

    Dongbin Zhang, Chuming Wang, Weitao Wang, Peihao Li, Minghan Qin, and Haoqian Wang. Gaussian in the wild: 3d gaussian splatting for unconstrained image collections. arXiv preprint arXiv:2403.15704, 2024 a

    2024 arXiv

  54. [63]

    Monst3r: A simple approach for estimating geometry in the presence of motion

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arXiv:2410.03825, 2024 b

    arXiv

  55. [64]

    Stereo Magnification: Learning View Synthesis using Multiplane Images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo Magnification: Learning View Synthesis using Multiplane Images . SIGGRAPH, 2018

    2018

  56. [65]

    Long-lrm: Long-sequence large reconstruction model for wide-coverage gaussian splats

    Chen Ziwen, Hao Tan, Kai Zhang, Sai Bi, Fujun Luan, Yicong Hong, Li Fuxin, and Zexiang Xu. Long-lrm: Long-sequence large reconstruction model for wide-coverage gaussian splats. arXiv preprint arXiv:2410.12781, 2024

    2024 arXiv

  57. [66]

    Shape of Motion: 4D Reconstruction from a Single Video

    Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of Motion: 4D Reconstruction from a Single Video . arXiv:2407.13764, 2024

    2024

  58. [67]

    Track anything: Segment anything meets videos

    Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. Track anything: Segment anything meets videos. arXiv preprint arXiv:2304.11968, 2023

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.