Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

OmniX claims that one frozen 2D flow-matching model, fitted with lightweight LoRA adapters, can generate and perceive panoramas and assemble them into relightable, graphics-ready 3D scenes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:03 UTC pith:3KFOKJM2

load-bearing objection A useful unified-adapter recipe and a new synthetic panorama dataset, wrapped in a 'graphics-ready' claim that outruns the evidence; worth refereeing, but the authors should release artifacts and test on real scenes. the 4 major comments →

arxiv 2510.26800 v2 pith:3KFOKJM2 submitted 2025-10-30 cs.CV cs.GRcs.LG

OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes

Yukun Huang , Jiwen Yu , Yanning Zhou , Jianan Wang , Xintao Wang , Pengfei Wan , Xihui Liu This is my paper
classification cs.CV cs.GRcs.LG
keywords panorama perceptioninverse renderingPBR material estimation3D scene generationflow matchingLoRA adapterspanoramic datasetgraphics-ready scenes
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

OmniX claims that a pre-trained 2D flow-matching image generator can be converted, without retraining its core weights, into a unified engine for panoramic image generation, intrinsic perception (distance, normals, albedo, roughness, metallic), and masked completion. The key mechanism is a Separate-Adapter cross-modal structure in which each input and output modality gets its own lightweight LoRA adapter while sharing the frozen diffusion transformer. If correct, a single-model chain from one perspective image to full panorama, to multimodal property maps, to a textured, relightable 3D mesh becomes practical, closing the gap between 2D lifting and graphics pipelines. The authors train twelve adapters on a new synthetic panorama dataset (PanoX) and report consistent gains over prior intrinsic-decomposition and geometry estimators on out-of-domain panoramas, then demonstrate PBR relighting and physics simulation in Blender.

Core claim

At its core, the paper tries to establish that perception of scene intrinsics—geometry and PBR materials—can be formulated as just another conditioning/generation task for a flow-matching model. Treating RGB-to-X as generation of X from an RGB condition (and optionally camera rays), and masked completion as generation from masked conditions, the same pre-trained 2D generative prior can be reused across all tasks. The load-bearing design choice is Separate-Adapter: distinct LoRA adapters for each condition and target, joined through cross-modal attention, which preserves the original weight distribution and generalizes better than channel-wise or token-wise sharing. The paper's evidence is qu

What carries the argument

Separate-Adapter cross-modal structure built on a pre-trained DiT-based flow matching model: each spatially aligned condition (RGB panorama, camera ray, mask) and each target modality gets its own LoRA adapter, and the adapters interact through cross-modal attention inside the MMDiT blocks. This lets one frozen backbone handle image-to-panorama generation, RGB-to-X perception for five modalities, and completion/guided perception, with a unified multiple-input/multiple-output flow-matching loss. Horizontal blending is used to enforce equirectangular seam continuity.

Load-bearing premise

The central claim depends on training on synthetic panoramas rendered from only eight Unreal Engine scenes transferring to arbitrary real-world scenes—especially for metallic and roughness maps, where the authors concede the metallic adapter 'performs poorly in generalization.'

What would settle it

Run the OmniX metallic and roughness adapters on a diverse set of real indoor and outdoor panoramas with known PBR ground truth (or close-up material checks) and compare against the synthetic-only PanoX results: if metallic maps degrade sharply on non-synthetic scenes, as the paper's own limitation section suggests, the 'graphics-ready for arbitrary scenes' claim collapses to the synthetic domain.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A single frozen 2D generator plus lightweight adapters can replace separate depth, normal, albedo, roughness, and metallic estimators for panoramic inputs.
  • Image-to-panorama generation can be extended to output geometry and PBR material maps in the same framework, enabling textured 3D meshes without manual UV work.
  • Mask-guided completion allows progressive, interactive scene extension from a single viewpoint, moving toward explorable and even city-scale 3D worlds.
  • The PanoX dataset provides a new benchmark for panoramic intrinsic decomposition covering both indoor and outdoor scenes.
  • If the pipeline generalizes, artists and robotics simulators could go from one photo to a relightable, physics-ready scene asset automatically.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Separate-Adapter design suggests a scaling route: adding a new modality (e.g., semantics, lighting, or HDRI) may require training only one more LoRA rather than re-tuning the generator—an extensibility the paper does not explicitly test.
  • A direct test is to train or fine-tune the metallic adapter on procedurally varied synthetic materials or real panoramic PBR captures; the paper's limitation section implies the bottleneck is data scarcity, not architecture.
  • Because the method avoids panorama-specific inductive biases apart from horizontal blending, the same adapter scheme may transfer to narrow field-of-view images; the paper notes this possibility but does not demonstrate it.
  • The bumpy surfaces caused by imperfect distance predictions could likely be smoothed by post-hoc mesh refinement or a depth-refinement network before assigning PBR materials—an integration the paper leaves implicit.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper introduces OmniX, a framework that adapts the pre-trained 2D flow-matching DiT model FLUX.1-dev to panoramic vision tasks via task-specific LoRA adapters and a Separate-Adapter cross-modal attention design. The proposed formulation covers image-to-panorama generation, RGB-to-X perception (distance, normal, albedo, roughness, metallic), and masked completion. The authors also construct PanoX, a synthetic Unreal Engine 5 dataset of 10k instances from 8 indoor/outdoor scenes with dense geometry and material annotations. Experiments on the held-out PanoX-OutDomain show large improvements over several baselines on albedo/roughness/metallic estimation and competitive depth/normal accuracy; qualitative results demonstrate panorama generation, completion, and downstream mesh-based relighting and physical simulation. The paper claims that this pipeline yields graphics-ready 3D scenes for PBR rendering, relighting, and simulation.

Significance. The perception results are promising: Table 2 shows consistent and large PSNR/LPIPS gains over five strong baselines on the held-out synthetic domain, and the ablations in Table 4 and Appendix D give useful evidence for the Separate-Adapter design and for independent rather than joint prediction. PanoX is a potentially valuable resource, being one of the few panoramic datasets with dense material annotations, and the paper is transparent about its limitations in Sec. E. However, the system-level headline claim—that OmniX enables graphics-ready 3D scenes for arbitrary input panoramas—is not supported by the current evidence: the quantitative evaluation is restricted to two synthetic scenes generated by the same UE5 pipeline as the training data, and the authors themselves concede that metallic prediction generalizes poorly. The strength of the paper is therefore in the method and dataset for panoramic perception, not yet in the demonstrated generality of the PBR/scene-construction application.

major comments (4)
  1. [§3.2, Sec. E, Tables 2–3] The only quantitative out-of-distribution test is PanoX-OutDomain, which consists of two scenes rendered from the same Unreal Engine 5 asset pipeline used to build the training scenes. This tests generalization to novel synthetic environments, not to real panoramic captures. Since the abstract and Sec. 3.4 claim graphics-ready 3D scenes for PBR rendering and relighting, the PBR material maps are load-bearing, yet Sec. E states that OmniX-Pano2Metallic 'performs poorly in generalization' and that 2D priors may have 'limited benefits for PBR material estimation.' Figure 9 provides only qualitative evidence on real images. Please add quantitative evaluation on real panoramic inputs (or a clearly justified proxy), or explicitly restrict the graphics-ready claim to synthetic and same-pipeline inputs.
  2. [Tables 2, 3, 4, 5, 6, 7] No error bars, confidence intervals, or significance tests are reported; all tables appear to be point estimates from a single training run. This matters especially because PanoX-OutDomain contains only two scenes, so the variance of the reported metric could be large relative to the differences between methods. For example, Table 3 shows OmniX as second-best on distance and the normal columns are not available for two baselines, making the comparison fragile. Please report standard deviations over multiple runs (or bootstrap over the test set) and state the number of seeds for the main comparisons.
  3. [Sec. 4.1, Tables 6–7, Sec. 4.5] Each modality is trained with an independent adapter, and the ablations show that joint modeling of geometry or materials does not improve accuracy. While this is a defensible design choice, it leaves open the question of cross-modal consistency: nothing ensures that the separately predicted distance, normal, albedo, roughness, and metallic maps correspond to the same underlying scene. For the 'graphics-ready' claim, this consistency is essential, because inconsistent maps will produce incorrect relighting and simulation. The only downstream validation (Sec. 4.5, Fig. 8) is qualitative. Please add a quantitative evaluation of the assembled 3D scene—for example, re-rendering error against PanoX ground truth, or multi-view mesh consistency—or explicitly soften the claim to 'visually plausible' rather than physically consistent.
  4. [Appendix B, Fig. 10] The qualitative image-to-panorama results in Fig. 10 are obtained from input images that, per the caption, are themselves generated by FLUX.1-dev. This does not demonstrate generalization to real single-view photographs, which is the practical setting for the claimed application. Please provide examples with real images, or state clearly that the generation results are limited to synthetic/generated inputs.
minor comments (5)
  1. [Abstract and Sec. 4.3] Typos: 'two prevalent ways to constructing' should be 'two prevalent ways of constructing'; 'we achieve the highest normal estimation accuracy and and the second highest' contains a doubled 'and'.
  2. [General] PanoX is presented as a contribution and benchmark, but no dataset/code release link or license is provided. Please add an availability statement.
  3. [Table 5] The column header suggests PSNR and LPIPS are reported for all modalities, but the roughness and metallic columns contain only one metric. Align the header with the reported entries or add the missing values.
  4. [References] Raistrick et al. 2023a and 2023b are identical; Yu et al. 2024c and 2024d are also identical. Cite each work once to avoid confusion.
  5. [Sec. 4.5] The demonstration videos are said to be on the project page, but no URL is given in the manuscript. Include the project page link in the final version.

Circularity Check

0 steps flagged

No material circularity: predictions are direct regression outputs, the 3D projection is deterministic, and the few self-citations are not load-bearing.

full rationale

The derivation chain is not circular. OmniX's unified formulation (Eqs. 1-5) extends the standard flow-matching objective to multiple conditional inputs and target modalities; the perception maps (distance, normal, albedo, roughness, metallic) are direct regression outputs of LoRA adapters trained on the PanoX synthetic dataset, not fitted parameters renamed as predictions. The loss in Eq. 4 is the standard flow-matching loss, and no fitted quantity is recycled as an evaluation target. The geometry-to-mesh stage in Sec. 3.4 is a deterministic transform: given the panoramic distance map and known ray directions, pixels are projected as mesh vertices and connectivity follows from pixel neighbors and relative distances; the citation to DreamCube is for this standard, independently implementable projection, so it is not load-bearing. The horizontal-blending choice follows LayerPano3D and is an engineering detail, not a definition of the claimed result. Evaluations compare against external methods on PanoX-OutDomain and Structured3D, providing independent grounding. The admitted limitation in Sec. E that OmniX-Pano2Metallic generalizes poorly weakens the strength of the graphics-ready claim for in-the-wild inputs, but this is a correctness/generalization concern, not circularity. No step reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim is an empirical system, so the ledger records the hand-set design choices and unproven transfer assumptions on which the headline results depend. The method adds no new physical or theoretical entities; the PanoX dataset is a dataset artifact, not a postulated entity.

free parameters (4)
  • LoRA ranks and adapter insertion points = not reported
    The capacity and placement of each task adapter control how much of the pretrained FLUX prior is reused; without this, results cannot be reproduced (Sec 3.3, Fig 4).
  • PanoX scene partition = 5 indoor + 3 outdoor; out-domain = 2 scenes
    All quantitative generalization claims rest on this hand-chosen split; results may shift with different scene assignments (Sec 3.2).
  • Training data mixture weights = equal probability PanoX/Structured3D
    The relative sampling of synthetic datasets is a hand-set hyperparameter affecting perception quality (Sec 4.2).
  • Occlusion-aware mask sampling parameters = not specified
    The mask generation for completion training depends on random 3D displacement and ray intersection thresholds that are not fully specified (Fig 5).
axioms (5)
  • domain assumption Flow matching loss with LoRA adapters preserves the base model's generative prior and is a valid training objective.
    The entire method is a fine-tuning recipe; there is no proof that LoRA adapters won't distort the pretrained distribution (Sec 3.3, Eq 4-5).
  • domain assumption Synthetic PanoX data is representative of real indoor/outdoor scenes.
    Used to train all perception adapters; real-world results are only qualitative (Sec 3.2, Fig 9).
  • ad hoc to paper Equirectangular seam continuity can be restored by horizontal blending from LayerPano3D.
    Introduced to fix topological limitations of 2D positional encoding; not derived from first principles (Sec 3.3 Remarks).
  • domain assumption Projecting a predicted panoramic distance map to vertices with pixel-neighbor connectivity yields a valid mesh for PBR texturing.
    Underlies 'graphics-ready' claim; no mesh quality evaluation (Sec 3.4).
  • domain assumption The RGB VAE latent space of FLUX.1-dev can represent single-channel PBR maps via cross-attention without a dedicated VAE.
    Motivates cross-attention material modeling; observed empirically but not proven (Supp D).

pith-pipeline@v1.3.0-alltime-deepseek · 14878 in / 13891 out tokens · 117722 ms · 2026-08-04T07:03:32.190525+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes." pith.science (2026). https://pith.science/paper/3KFOKJM2

@misc{pith2026251026800,
  author       = {Pith},
  title        = {Pith review of: OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3KFOKJM2}},
  note         = {Machine review of arXiv:2510.26800}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

There are two prevalent ways for automatic 3D scene construction: procedural generation and 2D lifting. Among these, panorama-based 2D lifting has emerged as a promising technique, leveraging powerful 2D generative priors to produce immersive, realistic, and diverse 3D environments. In this work, we advance this technique to generate graphics-ready 3D scenes suitable for physically based rendering (PBR), relighting, and simulation. Our key insight is to repurpose 2D generative models for panorama perception of geometry, textures, and PBR materials. Unlike existing 2D lifting approaches that emphasize appearance generation and neglect the perception of intrinsic properties, we present OmniX, a versatile and unified framework for panorama generation, perception, and completion. Built upon cross-modal adapter structure and cyclic spatial operators, OmniX effectively repurposes pre-trained 2D flow matching priors for joint modeling of multimodal, seamless equirectangular representations. Furthermore, we construct a large-scale synthetic panorama dataset comprising high-quality multimodal panoramas from diverse indoor and outdoor scenes. Extensive experiments demonstrate the effectiveness and generality of OmniX as a unified framework for panorama generation and perception across geometry, lighting, and semantics, enabling graphics-ready 3D scene generation and opening new possibilities for immersive and physically realistic virtual world creation.

Figures

Figures reproduced from arXiv: 2510.26800 by Jianan Wang, Jiwen Yu, Pengfei Wan, Xihui Liu, Xintao Wang, Yanning Zhou, Yukun Huang.

Figure 1
Figure 1. Figure 1: We present OmniX, a versatile and unified framework that repurposes pre-trained 2D flow matching models for panoramic perception, generation, and completion. This framework en￾ables the construction of immersive, photorealistic, and graphics-compatible 3D scenes, suitable for physically-based rendering (PBR), relighting, and physical dynamics simulation. ABSTRACT There are two prevalent ways to constructin… view at source ↗
Figure 2
Figure 2. Figure 2: A preview of the proposed PanoX dataset, providing high-quality panoramic rendered [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: OmniX pipeline for panoramic generation and perception. Built on a pre-trained 2D flow matching model with flexible, modality-specific adapters, OmniX is capable of performing a wide range of panoramic vision tasks including generation, perception, and completion. generalize the model fθ to take multiple condition inputs: zˆ1 = z0 + Z 1 0 fθ(zt, c 0 , c 1 , ..., y, t)dt, (3) where {c i |i = 0, 1, ...} are … view at source ↗
Figure 4
Figure 4. Figure 4: Different cross-modal adapter structures for multiple condition inputs {c i | i = 0, 1, ...} and multiple target outputs {ˆzj 1 | j = 0, 1, ...}. Specifically, (a) Shared-Branch concatenates differ￾ent inputs along the channel dimension; (b) Shared-Adapter is equivalent to token-wise concatena￾tion; (c) Separate-Adapter learns specific adapter weights for each type of input. 5 [PITH_FULL_IMAGE:figures/ful… view at source ↗
Figure 5
Figure 5. Figure 5: Occlusion-aware mask sampling. Based on the panoramic distance map and a randomly sampled 3D displacement, we can esti￾mate the occluded regions by ray intersection. These re￾gions are used as masks for training panoramic comple￾tion and guided panoramic perception models. Random Offset Distance Map Panorama Masked Panorama Occlusion-Aware Mask 6 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative evaluation of OmniX on panoramic intrinsic decomposition compared to state-of-the-art methods: IID (Kocsis et al., 2024), and DiffusionRenderer (Liang et al., 2025). Panoramic geometry estimation. We compare our OmniX with two panoramic geometry esti￾mation methods: DepthAnyCamera (Guo et al., 2025), DepthAnywhere Wang & Liu (2024), and four narrow-FoV geometry estimation methods: OmniData-v2 (… view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative evaluation of OmniX on panoramic geometry estimation compared to state-of-the-art geometry estimation methods: DepthAnyCamera (Guo et al., 2025), DepthAny￾where (Wang & Liu, 2024), OmniData-v2 (Kar et al., 2022), MGNet (Zhu et al., 2022), Diffu￾sionRenderer (Liang et al., 2025), and MoGe (Wang et al., 2025). Our approach demonstrates a notable advantage in capturing fine image details, thanks t… view at source ↗
Figure 8
Figure 8. Figure 8: Demonstrations of the graphics-compatible 3D scenes created with OmniX, ready for free exploration, PBR-based relighting, and physical dynamics simulation. 5 CONCLUSION In this work, we present OmniX, a versatile framework for repurposing pre-trained 2D flow match￾ing models for panoramic perception, generation, and completion. Specifically, we establish a unified formulation that incorporates dense visual… view at source ↗
Figure 10
Figure 10. Figure 10: Our method is able to achieve high-quality and diverse image-to-panorama generation. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 9
Figure 9. Figure 9: Panoramic perception results of OmniX on in-the-wild images. Our method demonstrates [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Panorama generation results of OmniX given a single image input. Note that the input [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Panorama completion and guided panoramic perception results of OmniX. Given masked [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rein3D: Reinforced 3D Indoor Scene Generation with Panoramic Video Diffusion Models

    cs.CV 2026-04 unverdicted novelty 6.0

    Rein3D generates photorealistic, globally consistent 3D indoor scenes by using a restore-and-refine process where radial panoramic videos are restored via diffusion models and then used to update a 3D Gaussian field.

Reference graph

Works this paper leans on

68 extracted references · 6 linked inside Pith · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Rethinking Inductive Biases for Surface Normal Estimation

    Gwangbin Bae and Andrew J Davison. Rethinking Inductive Biases for Surface Normal Estimation . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 9535--9545, 2024

    2024

  3. [3]

    Shape, illumination, and reflectance from shading

    Jonathan T Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. IEEE transactions on pattern analysis and machine intelligence, 37 0 (8): 0 1670--1687, 2014

    2014

  4. [4]

    Recovering intrinsic scene characteristics

    Harry Barrow, J Tenenbaum, A Hanson, and E Riseman. Recovering intrinsic scene characteristics. Comput. vis. syst, 2 0 (3-26): 0 2, 1978

    1978

  5. [5]

    Intrinsic images in the wild

    Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM Transactions on Graphics (TOG), 33 0 (4): 0 1--12, 2014

    2014

  6. [6]

    Stylegan knows normal, depth, albedo, and more

    Anand Bhattad, Daniel McKee, Derek Hoiem, and David Forsyth. Stylegan knows normal, depth, albedo, and more. Advances in Neural Information Processing Systems, 36: 0 73082--73103, 2023

    2023

  7. [7]

    User-assisted intrinsic images

    Adrien Bousseau, Sylvain Paris, and Fr \'e do Durand. User-assisted intrinsic images. In ACM SIGGRAPH Asia, pp.\ 1--10, 2009

    2009

  8. [8]

    Text2light: Zero-shot text-driven hdr panorama generation

    Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. Text2light: Zero-shot text-driven hdr panorama generation. ACM Transactions on Graphics (TOG), 41 0 (6): 0 1--16, 2022

    2022

  9. [9]

    Depth estimation via affinity learned with convolutional spatial propagation network

    Xinjing Cheng, Peng Wang, and Ruigang Yang. Depth estimation via affinity learned with convolutional spatial propagation network. In Proceedings of the European conference on computer vision (ECCV), pp.\ 103--119, 2018

    2018

  10. [10]

    Authoring landscapes by combining ecosystem and terrain erosion simulation

    Guillaume Cordonnier, Eric Galin, James Gain, Bedrich Benes, Eric Gu \'e rin, Adrien Peytavie, and Marie-Paule Cani. Authoring landscapes by combining ecosystem and terrain erosion simulation. ACM Transactions on Graphics (TOG), 36 0 (4): 0 1--12, 2017

    2017

  11. [11]

    Automated creation of digital cousins for robust policy learning

    Tianyuan Dai, Josiah Wong, Yunfan Jiang, Chen Wang, Cem Gokmen, Ruohan Zhang, Jiajun Wu, and Li Fei-Fei. Automated creation of digital cousins for robust policy learning. arXiv preprint arXiv:2410.07408, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Guided co-modulated gan for 360 field of view extrapolation

    Mohammad Reza Karimi Dastjerdi, Yannick Hold-Geoffroy, Jonathan Eisenmann, Siavash Khodadadeh, and Jean-Fran c ois Lalonde. Guided co-modulated gan for 360 field of view extrapolation. In 2022 International Conference on 3D Vision (3DV), pp.\ 475--485. IEEE, 2022

    2022

  13. [13]

    Procthor: Large-scale embodied ai using procedural generation

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35: 0 5982--5994, 2022

    2022

  14. [14]

    Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M \"u ller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis . In International Conference on Learning Representations, 2024

    2024

  15. [15]

    Layoutgpt: Compositional visual planning and generation with large language models

    Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 36: 0 18225--18250, 2023

    2023

  16. [16]

    Ground truth dataset and baseline evaluations for intrinsic image algorithms

    Roger Grosse, Micah K Johnson, Edward H Adelson, and William T Freeman. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In 2009 IEEE 12th International Conference on Computer Vision, pp.\ 2335--2342. IEEE, 2009

    2009

  17. [17]

    Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera

    Yuliang Guo, Sparsh Garg, S Mahdi H Miangoleh, Xinyu Huang, and Liu Ren. Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera . In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 26996--27006, 2025

    2025

  18. [18]

    Text2room: Extracting textured 3d meshes from 2d text-to-image models

    Lukas H \"o llein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nie ner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 7909--7920, 2023

    2023

  19. [19]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models . In International Conference on Learning Representations, 2022

    2022

  20. [20]

    DreamCube: 3D Panorama Generation via Multi-plane Synchronization

    Yukun Huang, Yanning Zhou, Jianan Wang, Kaiyi Huang, and Xihui Liu. DreamCube: 3D Panorama Generation via Multi-plane Synchronization . arXiv preprint arXiv:2506.17206, 2025 a

    Pith/arXiv arXiv 2025

  21. [21]

    Litereality: Graphics-ready 3d scene reconstruction from rgb-d scans

    Zhening Huang, Xiaoyang Wu, Fangcheng Zhong, Hengshuang Zhao, Matthias Nie ner, and Joan Lasenby. Litereality: Graphics-ready 3d scene reconstruction from rgb-d scans. arXiv preprint arXiv:2507.02861, 2025 b

    arXiv 2025

  22. [22]

    3d common corruptions and data augmentation

    O g uzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 18963--18974, 2022

    2022

  23. [23]

    Intrinsic Image Diffusion for Indoor Single-view Material Estimation

    Peter Kocsis, Vincent Sitzmann, and Matthias Nie ner. Intrinsic Image Diffusion for Indoor Single-view Material Estimation . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 5198--5208, 2024

    2024

  24. [24]

    IntrinsiX: High-Quality PBR Generation using Image Priors

    Peter Kocsis, Lukas H \"o llein, and Matthias Nie ner. IntrinsiX: High-Quality PBR Generation using Image Priors . In Advances in Neural Information Processing Systems, 2025

    2025

  25. [25]

    Flux.1-dev

    Black Forest Labs. Flux.1-dev. https://huggingface.co/black-forest-labs/FLUX.1-dev, 2025. Accessed: 2025-01-19

    2025

  26. [26]

    Vividdream: Generating 3d scene with ambient dynamics

    Yao-Chih Lee, Yi-Ting Chen, Andrew Wang, Ting-Hsuan Liao, Brandon Y Feng, and Jia-Bin Huang. Vividdream: Generating 3d scene with ambient dynamics. arXiv preprint arXiv:2405.20334, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023 a

    2023

  28. [28]

    4k4dgen: Panoramic 4d generation at 4k resolution

    Renjie Li, Panwang Pan, Bangbang Yang, Dejia Xu, Shijie Zhou, Xuanyang Zhang, Zeming Li, Achuta Kadambi, Zhangyang Wang, Zhengzhong Tu, et al. 4k4dgen: Panoramic 4d generation at 4k resolution. arXiv preprint arXiv:2406.13527, 2024 a

    Pith/arXiv arXiv 2024

  29. [29]

    InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset

    Wenbin Li, Sajad Saeedi, John McCormac, Ronald Clark, Dimos Tzoumanikas, Qing Ye, Yuzhong Huang, Rui Tang, and Stefan Leutenegger. InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset . In British Machine Vision Conference (BMVC), 2018

    2018

  30. [30]

    SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting

    Wenrui Li, Fucheng Cai, Yapeng Mi, Zhe Yang, Wangmeng Zuo, Xingtao Wang, and Xiaopeng Fan. SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting . arXiv preprint arXiv:2408.13711, 2024 b

    Pith/arXiv arXiv 2024

  31. [31]

    3d cinemagraphy from a single image

    Xingyi Li, Zhiguo Cao, Huiqiang Sun, Jianming Zhang, Ke Xian, and Guosheng Lin. 3d cinemagraphy from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4595--4605, 2023 b

    2023

  32. [32]

    MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond

    Yixuan Li, Lihan Jiang, Linning Xu, Yuanbo Xiangli, Zhenzhi Wang, Dahua Lin, and Bo Dai. MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond . In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 3205--3215, 2023 c

    2023

  33. [33]

    PhyIR: Physics-Based Inverse Rendering for Panoramic Indoor Images

    Zhen Li, Lingli Wang, Xiang Huang, Cihui Pan, and Jiaqi Yang. PhyIR: Physics-Based Inverse Rendering for Panoramic Indoor Images . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 12713--12723, 2022

    2022

  34. [34]

    Cgintrinsics: Better intrinsic image decomposition through physically-based rendering

    Zhengqi Li and Noah Snavely. Cgintrinsics: Better intrinsic image decomposition through physically-based rendering. In Proceedings of the European conference on computer vision (ECCV), pp.\ 371--387, 2018

    2018

  35. [35]

    Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image

    Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 2475--2484, 2020

    2020

  36. [36]

    IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations

    Zhibing Li, Tong Wu, Jing Tan, Mengchen Zhang, Jiaqi Wang, and Dahua Lin. IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations . In International Conference on Learning Representations, 2025

    2025

  37. [37]

    Envidr: Implicit differentiable renderer with neural environment lighting

    Ruofan Liang, Huiting Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijaykumar. Envidr: Implicit differentiable renderer with neural environment lighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 79--89, 2023

    2023

  38. [38]

    Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models

    Ruofan Liang, Zan Gojcic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Chih-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, et al. Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models . In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 26069--26080, 2025

    2025

  39. [39]

    Flow Matching for Generative Modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow Matching for Generative Modeling . In International Conference on Learning Representations, 2023

    2023

  40. [40]

    The synthesis and rendering of eroded fractal terrains

    F Kenton Musgrave, Craig E Kolb, and Robert S Mace. The synthesis and rendering of eroded fractal terrains. ACM Siggraph Computer Graphics, 23 0 (3): 0 41--50, 1989

    1989

  41. [41]

    Procedural modeling of cities

    Yoav IH Parish and Pascal M \"u ller. Procedural modeling of cities. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp.\ 301--308, 2001

    2001

  42. [42]

    Scalable Diffusion Models with Transformers

    William Peebles and Saining Xie. Scalable Diffusion Models with Transformers . In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 4195--4205, 2023

    2023

  43. [43]

    Infinite photorealistic worlds using procedural generation

    Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, et al. Infinite photorealistic worlds using procedural generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12630--12641, 2023 a

    2023

  44. [44]

    Infinite photorealistic worlds using procedural generation

    Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, et al. Infinite photorealistic worlds using procedural generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12630--12641, 2023 b

    2023

  45. [45]

    Susskind

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim : A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding . In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 10912--10922, 2021

    2021

  46. [46]

    Neural inverse rendering of an indoor scene from a single image

    Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W Jacobs, and Jan Kautz. Neural inverse rendering of an indoor scene from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 8598--8607, 2019

    2019

  47. [47]

    MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion

    Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion . In Advances in Neural Information Processing Systems, 2023

    2023

  48. [48]

    Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation

    Ning-Hsu Albert Wang and Yu-Lun Liu. Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation . Advances in Neural Information Processing Systems, 37: 0 127739--127764, 2024

    2024

  49. [49]

    MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision

    Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision . In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 5261--5271, 2025

    2025

  50. [50]

    Learning indoor inverse rendering with 3d spatially-varying lighting

    Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz. Learning indoor inverse rendering with 3d spatially-varying lighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 12538--12547, 2021

    2021

  51. [51]

    De-rendering 3d objects in the wild

    Felix Wimbauer, Shangzhe Wu, and Christian Rupprecht. De-rendering 3d objects in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 18490--18499, 2022

    2022

  52. [52]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4818--4829, 2024

    2024

  53. [53]

    LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation

    Shuai Yang, Jing Tan, Mengchen Zhang, Tong Wu, Gordon Wetzstein, Ziwei Liu, and Dahua Lin. LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation . In SIGGRAPH, pp.\ 1--10, 2025

    2025

  54. [54]

    4real: Towards photorealistic 4d scene generation via video diffusion models

    Heng Yu, Chaoyang Wang, Peiye Zhuang, Willi Menapace, Aliaksandr Siarohin, Junli Cao, L \'a szl \'o Jeni, Sergey Tulyakov, and Hsin-Ying Lee. 4real: Towards photorealistic 4d scene generation via video diffusion models. Advances in Neural Information Processing Systems, 37: 0 45256--45280, 2024 a

    2024

  55. [55]

    Wonderworld: Interactive 3d scene generation from a single image

    Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William T Freeman, and Jiajun Wu. Wonderworld: Interactive 3d scene generation from a single image. arXiv preprint arXiv:2406.09394, 2024 b

    Pith/arXiv arXiv 2024

  56. [56]

    Wonderjourney: Going from anywhere to everywhere

    Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T Freeman, Forrester Cole, Deqing Sun, Noah Snavely, Jiajun Wu, et al. Wonderjourney: Going from anywhere to everywhere. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 6658--6667, 2024 c

    2024

  57. [57]

    Wonderjourney: Going from anywhere to everywhere

    Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T Freeman, Forrester Cole, Deqing Sun, Noah Snavely, Jiajun Wu, et al. Wonderjourney: Going from anywhere to everywhere. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 6658--6667, 2024 d

    2024

  58. [58]

    Make it home: automatic optimization of furniture arrangement

    Lap-Fai Yu, Sai Kit Yeung, Chi-Keung Tang, Demetri Terzopoulos, Tony F Chan, and Stanley J Osher. Make it home: automatic optimization of furniture arrangement. ACM Trans. Graph., 30 0 (4): 0 86, 2011

    2011

  59. [59]

    RGB X: Image Decomposition and Synthesis Using Material- and Lighting-aware Diffusion Models

    Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling-Qi Yan, and Milo s Ha s an. RGB X: Image Decomposition and Synthesis Using Material- and Lighting-aware Diffusion Models . In ACM SIGGRAPH, 2024

    2024

  60. [60]

    Taming stable diffusion for text to 360 panorama image generation

    Cheng Zhang, Qianyi Wu, Camilo Cruz Gambardella, Xiaoshui Huang, Dinh Phung, Wanli Ouyang, and Jianfei Cai. Taming stable diffusion for text to 360 panorama image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 6347--6357, 2024 a

    2024

  61. [61]

    Text2nerf: Text-driven 3d scene generation with neural radiance fields

    Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d scene generation with neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 30 0 (12): 0 7749--7762, 2024 b

    2024

  62. [62]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

    2018

  63. [63]

    Structured3D: A Large Photo-Realistic Dataset for Structured 3D Modeling

    Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3D: A Large Photo-Realistic Dataset for Structured 3D Modeling . In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part IX 16, pp.\ 519--535. Springer, 2020

    2020

  64. [64]

    Dreamscene360: Unconstrained text-to-3d scene generation with panoramic gaussian splatting

    Shijie Zhou, Zhiwen Fan, Dejia Xu, Haoran Chang, Pradyumna Chari, Tejas Bharadwaj, Suya You, Zhangyang Wang, and Achuta Kadambi. Dreamscene360: Unconstrained text-to-3d scene generation with panoramic gaussian splatting. In European Conference on Computer Vision, pp.\ 324--342. Springer, 2024

    2024

  65. [65]

    Learning-Based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing

    Jingsen Zhu, Fujun Luan, Yuchi Huo, Zihao Lin, Zhihua Zhong, Dianbing Xi, Rui Wang, Hujun Bao, Jiaxiang Zheng, and Rui Tang. Learning-Based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing . In ACM SIGGRAPH Asia, 2022

    2022

  66. [66]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  67. [67]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  68. [68]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv