Pith. sign in

REVIEW 3 major objections 4 minor 73 references

JointDiT: Enhancing RGB-Depth Joint Modeling with Diffusion Transformers

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read JointDiT models the joint distribution of RGB images and depth maps so that joint generation, depth estimation, and depth-conditioned image generation all become special cases of one flow-matching network controlled by two timesteps.

desk verdict Solid empirical paper extending joint RGB-depth diffusion to a Flux transformer with two simple, ablated techniques; the 'viable alternative' claim outruns the evidence because the comparison mixes base models. read the letter →

arxiv 2505.00482 v3 pith:Q5TXP4KL submitted 2025-05-01 cs.CV cs.AI

Kwon Byung-Ki , Qi Dai , Lee Hyoseok , Chong Luo , Tae-Hyun Oh This is my paper · ORCID
classification cs.CVcs.AI
keywords jointdistributionmodelingdiffusiontransformerRGB-Ddepthestimationdepth-conditionedimagegenerationflowmatchingadaptiveschedulingweightstimestepsampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that modeling the joint distribution of color images and depth maps is enough to solve three different tasks with a single network: generating an image together with its depth, estimating depth from an image, and generating an image from a given depth map. The proposed model, JointDiT, appends a parallel depth branch to a pre-trained diffusion transformer and trains the two branches with separate noise timesteps, so the task is selected at inference time by setting the timestep pair. Two training techniques are introduced to make that work: adaptive scheduling weights that let the noisier branch borrow structure from the cleaner branch, and an unbalanced timestep sampling strategy that covers extreme noise combinations. If the central claim is right, task-specific heads and per-task fine-tuning pipelines for depth estimation could be replaced by one shared generative model, which matters because depth estimation and depth-conditioned image generation are currently built as separate conditional models.

What carries the argument

The engine is a two-timestep flow-matching objective over paired modalities. Instead of one diffusion time, JointDiT samples independent timesteps $t_x$ and $t_y$ for the RGB and depth branches and minimizes the Joint Conditional Flow Matching loss, which forces the network to predict both components of the joint vector field at every noise-level combination. The joint connection module is a cross-attention layer between branches whose outputs are scaled by adaptive scheduling weights $w_x(t_x,t_y)$ and $w_y(t_x,t_y)$; each weight is a sigmoid of the relative noise imbalance, so the cleaner branch's structure is transferred more strongly to the noisier branch. The unbalanced timestep sampling strategy draws $t_x$ and $t_y$ independently from two different timestep distributions half of the time, and from the same distribution the other half, ensuring coverage of the extreme pairs needed for the conditional tasks.

What would settle it

Replace the teacher-generated depth labels with ground-truth depth from the synthetic datasets at their original scale, retrain the same JointDiT pipeline, and compare absolute relative error on ETH3D; if the gap against the diffusion-based depth estimators does not shrink, then the paper's attribution of its ETH3D weakness to the teacher is not the explanation.

Watch

Extended reading notes

Core claim

JointDiT's central claim is that a single flow-matching network can learn the joint distribution of an RGB image and its depth map over every combination of their noise levels, and that the resulting model is a viable replacement for conditional generation in depth tasks. The training objective is a joint conditional flow-matching loss with two independent timesteps, $t_x$ for the image and $t_y$ for the depth, so the vector field $v_{t_x,t_y}(x,y|x_1,y_1)$ is learned for all pairs. At inference, $(t_x,t_y)=(0,0)$ yields joint image-depth generation, $(1,0)$ gives depth estimation from a clean image, and $(0,1)$ gives depth-conditioned image generation. The architecture keeps a pre-trained diffusion transformer's image branch frozen, adds a parallel depth branch through low-rank adapters, and exchanges features between branches through joint cross-attention modules gated by adaptive scheduling weights. The paper reports high-fidelity joint generation with geometrically plausible 3D lifting, depth-estimation accuracy comparable to dedicated generative depth estimators on several benchmarks, and better depth-conditioned generation scores than the conditional baselines compared on the same training data.

Load-bearing premise

The load-bearing premise is that the depth maps produced by the pre-trained teacher predictor, normalized to the unit interval, are faithful enough to serve as ground truth for the depth branch, so any systematic error in those labels is inherited by JointDiT.

Editorial extensions

If this is right

  • One frozen image-generation backbone plus trainable adapters can serve as a unified model for joint generation, monocular depth estimation, and depth-conditioned image generation, with the task selected by timesteps instead of by network head.
  • Depth estimation from a single image no longer needs a task-specific architecture: with only a small fraction of trainable parameters, the joint model matches diffusion-based depth estimators on NYUv2 and ScanNet and improves on them after a short fine-tuning on synthetic data.
  • Jointly generated image-depth pairs lift to more coherent 3D point clouds than depth estimated from the already-generated image, and this advantage extends to stylized domains such as cartoon and pixel-art images.
  • In depth-conditioned image generation, the joint model provides lower FID and lower depth-consistency error than the conditional baselines compared on the same data.
  • Feature analysis shows the depth branch encodes structure while the RGB branch encodes texture, suggesting the joint model can serve as a geometric prior rather than only an appearance generator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the real-image training pairs use teacher-generated depth labels, the teacher's systematic errors set a ceiling on the model; a better teacher or a mix of ground-truth and synthetic depth should improve the conditional tasks, and the paper's own ETH3D gap is consistent with this.
  • The same two-timestep recipe should transfer to other paired modalities where one branch is structurally reliable, such as normals, segmentation, or albedo, and could be tested with identical adaptive scheduling and unbalanced sampling.
  • The joint model's ability to run at varying resolutions and without a text prompt suggests it could serve as a prior for inverse problems like depth completion or inpainting, which the paper does not explore.
  • The observed RGB-texture / depth-structure division of labor could be turned into a test: if joint generation is used to synthesize depth for out-of-domain illustrations, it should generalize better than monocular depth estimators, which the paper partly demonstrates but does not quantify.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes JointDiT, a diffusion transformer that models the joint distribution of RGB images and depth maps. The architecture extends a frozen Flux backbone with a parallel depth branch using LoRA adapters and joint cross-attention modules. Two techniques are introduced: adaptive scheduling weights, which modulate cross-modal information transfer according to the relative noise levels of the two branches, and an unbalanced timestep sampling strategy, which trains on both shared and independent timesteps per modality. By controlling the timesteps of each branch, the same network performs joint RGB-depth generation, depth estimation from an image, and depth-conditioned image generation. Experiments report strong joint generation quality, depth estimation comparable to diffusion-based depth estimators, and state-of-the-art depth-conditioned generation on the UniCon evaluation protocol. The central claim is that joint distribution modeling can serve as a viable alternative to conditional generation.

Significance. If the central claim is established, the paper offers a unified framework where one network handles several tasks through timestep control, which is a conceptually appealing direction for multi-modal diffusion models. The architecture is described in detail, and the ablation studies in Section 4.4 provide credible internal evidence that the two proposed techniques improve joint generation and depth estimation. The paper is generally clearly written and includes a limitation statement. However, the headline claim that joint distribution modeling can replace conditional generation is not supported by the present comparisons: the depth-conditioned generation baselines use a different and weaker base model (Stable Diffusion) than JointDiT (Flux), and no same-backbone conditional control is provided. In addition, the depth estimation evaluation lacks a precise description of the alignment protocol, which is essential for interpreting the AbsRel numbers. These gaps are fixable within the scope of a revision.

major comments (3)
  1. [§4.3, Table 2; Abstract and Conclusion] The claim that joint distribution modeling is a viable alternative to conditional generation is not established because the comparison in Table 2 does not control for the base generative model or the joint pretraining stage. Readout-Guidance, ControlNet, and UniCon are all based on Stable Diffusion, whereas JointDiT uses the Flux transformer backbone, adds a parallel depth branch with LoRA, and is first joint-pretrained on 50k image-depth pairs before the PascalVOC fine-tuning described in Supplementary A.1. The lower FID and AbsRel of JointDiT may therefore be due to the stronger Flux prior and the architecture rather than to joint-distribution modeling. Please add a depth-conditioned image generator with the same Flux backbone and the same LoRA budget that does not use joint pretraining, or run an ablation of JointDiT without the 50k joint-pretraining stage. Without such a control, the paper should not claim that joint modeling itself is responsible for the improvement.
  2. [§4.2, Table 1; Supplementary A.1] The depth estimation evaluation protocol is underspecified regarding the alignment of predicted depth to ground truth. JointDiT outputs normalized disparity, yet the reported AbsRel values (e.g., 5.7 on NYUv2) are only meaningful if some affine alignment (e.g., least-squares or median) to the ground truth is applied. The paper states that no ensemble technique is used but does not state whether or how alignment was performed for JointDiT or for the compared methods Marigold and GeoWizard. Without a precise statement of the alignment procedure, the numbers in Table 1 are not reproducible and the comparison to generative depth estimators is not interpretable. Please specify the exact evaluation protocol, including the alignment step, for all methods.
  3. [§4.2 and Supplementary A.2] The depth branch is trained on Depth-Anything-V2 pseudo-labels, and the synthetic fine-tuning data are rescaled to match DA-2's mean and standard deviation. The paper acknowledges that this teacher leads to a weak ETH3D result (AbsRel 16.5 vs 6.4 for GeoWizard), but the claim of comparable depth estimation on NYUv2, ScanNet, KITTI, and DIODE is still conditional on the teacher's quality. Because systematic errors in DA-2 predictions are likely inherited by JointDiT, the paper should discuss the potential bias of this supervision more thoroughly and, if possible, report a variant trained with a different pseudo-label source or elevate the Ours+ft results (with synthetic ground truth) as the primary evidence for the depth-estimation capability.
minor comments (4)
  1. [§3.1, Eq. (6)] The adaptive scheduling weights wx and wy are undefined when tx + ty = 0, which corresponds to the initialization for joint generation. Please add a convention (e.g., define the value as 1/2 by limit) or add an epsilon to the denominator so the formula is well-defined for all inputs.
  2. [§4.3, Table 2 caption] The phrase "With the same training dataset" is potentially misleading because the compared methods also differ in base pretraining and in the joint-pretraining stage; please rephrase as "with the same fine-tuning dataset" or clarify the exact sense in which the training data are the same.
  3. [Table 4] In the depth-conditioned image generation ablation, the full model (both techniques) achieves FID 12.58, which is not the best among the four configurations (the "neither" configuration gives 11.94 and the "unbalanced only" configuration gives 12.51). The claim that the two techniques together are effective for this task rests primarily on the ImageReward ranking; please clarify why FID does not improve and whether human-preference ranking is the intended primary metric.
  4. [Supplementary A.3, Eq. (7)] The definition of f(t) is given, but g(t) is only described as f(t) with s=0.25; please write g(t) explicitly to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivational circularity: the joint-modeling pipeline trains on external pseudo-labels and is evaluated on independent benchmarks; no equation in the method reduces to its own outputs.

full rationale

The method section builds a two-timestep flow-matching objective (Eq. 4), adds a parallel depth branch with LoRAs and joint cross-attention from UniCon, and proposes adaptive scheduling weights (Eqs. 5-6) and unbalanced timestep sampling. None of these components is defined in terms of the evaluation metrics, and the test-time protocols (tx=1/ty=0 for depth estimation, tx=0/ty=1 for depth-conditioned generation) are direct evaluations of the trained vector field rather than re-statements of the training target. Depth-estimation numbers are reported on NYUv2, ScanNet, KITTI, DIODE, and ETH3D, none of which supplies the training signal; the paper explicitly attributes its ETH3D AbsRel weakness to the Depth-Anything-V2 teacher, a data-quality limitation rather than a circular reduction. Table 2 follows UniCon's protocol with the same PascalVOC training split for all methods and OpenImages for evaluation; the AbsRel there is a consistency check against an off-the-shelf depth model, so the metric could be self-referential if the conditioning and evaluation depth models coincide, but the paper fits no parameter to that metric and no equation forces the result. The self-citations (e.g., [28] on depth completion) appear only as Related Work context and are not load-bearing for the central joint-modeling claim. There is no imported uniqueness theorem, no ansatz smuggled in through a self-citation, and no renaming of a known empirical pattern as organization. The strongest caveat—the absence of a same-backbone conditional-generation baseline—is an experimental-confound concern, not a circularity, and is outside the circularity rubric.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central method rests on standard flow matching, the assumption that separate per-modality timesteps yield the conditional tasks, and the choice of Depth-Anything-V2 pseudo-labels as supervision. The main hand-set constants are alpha=3 and the parameters of the two timestep distributions; these affect training but are not fitted to the evaluation benchmarks.

free parameters (5)
  • alpha (adaptive scheduling weight scale) = 3
    Controls the slope of the sigmoid gating in Eq. 6. Set by hand for all experiments; sensitivity not reported.
  • s in f(t) = 3.1582
    Timestep sampling distribution from the base sd-scripts training code; not original to this paper, but the central training distribution depends on it.
  • s in g(t) = 0.25
    Chosen to make the unbalanced timestep distribution; no ablation or derivation is given for this value.
  • probability of separate vs shared timestep sampling = 0.5
    Half probability for sampling tx and ty from different distributions; set by design without sensitivity analysis.
  • LoRA rank = 64 (DiT blocks), 512/1024 (input stage)
    Selected after an ablation on NYUv2 and ScanNet (Table 8); affects capacity and results.
assumptions (6)
  • standard math Flow matching and conditional flow matching objectives from Lipman et al. are valid for training the joint vector field.
    Used in Section 2 and Eq. 4 without proof; standard background.
  • domain assumption Sampling separate timesteps tx and ty induces a joint flow that, at endpoints, yields the conditional densities needed for depth estimation and depth-conditioned generation.
    Section 3 states tasks are achieved by adjusting tx and ty. The training distributions f(t) and g(t) have support near but not necessarily at t=0 and t=1, and the paper provides no proof that the learned vector field at extreme timesteps equals the conditional target.
  • domain assumption Depth-Anything-V2 pseudo-labels are acceptable as ground-truth depth for training the joint model.
    Section 4 Implementation details: depth maps are generated by Depth-Anything-V2. This transfers the teacher's systematic errors to JointDiT.
  • ad hoc to paper Pre-trained Flux's image prior transfers to a depth branch through a frozen backbone plus LoRA.
    Core architectural assumption of Section 3.1; empirically supported by results but not derived.
  • domain assumption Joint cross-attention from UniCon is sufficient to align RGB and depth features.
    Adopted from prior work in Section 3.1 with no comparison to other fusion schemes.
  • ad hoc to paper The visual interpretation that the depth branch focuses on geometry and the RGB branch on texture explains the improved joint generation.
    Section 4.4 and Figure 6 present this as a post-hoc interpretation without causal verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JointDiT: Enhancing RGB-Depth Joint Modeling with Diffusion Transformers." pith.science (2026). https://pith.science/paper/Q5TXP4KL

@misc{pith2026250500482,
  author       = {Pith},
  title        = {Pith review of: JointDiT: Enhancing RGB-Depth Joint Modeling with Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q5TXP4KL}},
  note         = {Machine review of arXiv:2505.00482}
}
read the original abstract

We present JointDiT, a diffusion transformer that models the joint distribution of RGB and depth. By leveraging the architectural benefit and outstanding image prior of the state-of-the-art diffusion transformer, JointDiT not only generates high-fidelity images but also produces geometrically plausible and accurate depth maps. This solid joint distribution modeling is achieved through two simple yet effective techniques that we propose, namely, adaptive scheduling weights, which depend on the noise levels of each modality, and the unbalanced timestep sampling strategy. With these techniques, we train our model across all noise levels for each modality, enabling JointDiT to naturally handle various combinatorial generation tasks, including joint generation, depth estimation, and depth-conditioned image generation by simply controlling the timesteps of each branch. JointDiT demonstrates outstanding joint generation performance. Furthermore, it achieves comparable results in depth estimation and depth-conditioned image generation, suggesting that joint distribution modeling can serve as a viable alternative to conditional generation. The project page is available at https://byungki-k.github.io/JointDiT/.

Figures

Figures reproduced from arXiv: 2505.00482 by the authors.

Figure 1
Figure 1. We present JointDiT, a diffusion transformer modeling the RGB-Depth joint distribution. By leveraging the image prior of a [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. 3D lifting results of JointDiT and depth diffusion models, i.e., Marigold [32] and GeoWizard [21]. JointDiT also shows more plausible 3D point clouds than depth estimation models in challenging illustration domains, likely due to the complementary behavior of the RGB and depth branches across generative processes, i.e., the RGB branch focuses on texture, and the depth branch on structure. and even further cross-moda… view at source ↗
Figure 3
Figure 3. Overall pipeline of JointDiT. Building on Flux [7], we introduce a parallel depth branch with trainable LoRAs [27]. The joint connection module enables the aligned joint generation. We propose adaptive scheduling weights and an unbalanced timestep sampling strategy for effective training using separate timesteps. MM-DiT and P-DiT denote the multi-modal [19] and parallel [18] diffusion transformers, respectively. The… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: 3D lifting results of LDM3D [60], JointNet [71], and our JointDiT. Our JointDiT generates highly plausible image-aligned 3D structures, surpassing previous joint generation methods in achieving superior consistency with real 3D space. eration. We also analyze the behav…
Figure 5
Figure 5. Figure 5: Depth estimation results of joint generation models. We visualize the depth estimation results of JointNet, UniCon, and our method on the NYUv2 and ScanNet dataset. Our approach captures thin and fine-grained details with only a timestep adjust￾ment, i.e., tx = 1 and t…
Figure 6
Figure 6. Figure 6: Feature visualization of RGB-Depth branches in MM-DiT blocks. We observe that, at timestep t = 0.48, the depth branch tends to focus on scene geometry, while the RGB branch captures semantic patterns related to texture and appearance. generation from text prompts, even…
Figure 7
Figure 7. Figure 7: Comparison of 3D lifting results from our JointDiT, Marigold, and Depth-Anything-V2. The jointly generated depth from JointDiT leads to more coherent 3D shapes and better preservation of structural details compared to the estimated depths. Generation Method ImageNet 6K…
Figure 8
Figure 8. Figure 8: Comparison between original images and images generated by Flux [7] on the ImageNet [53] 6K dataset. Flux often generates stylized images, which leads to a higher FID be￾tween the real image dataset and the generated images. the limited size of the training dataset. Am…
Figure 9
Figure 9. Figure 9: Failure cases in depth estimation. Red and Blue areas indicate near and far depth predictions, respectively. B.5. Joint Panorama Generation JointDiT can be used for RGB-D panorama generation as well. For panorama generation, we strictly follow the Joint￾Net [71] method…
Figure 10
Figure 10. Figure 10 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Joint generation results of JointDiT. The joint generated images and depths are geometrically reasonable in 3D [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Joint generation results in illustration domains. The jointly generated images and depths from JointDiT produce geometri￾cally plausible 3D structures, even in stylized domains [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Depth estimation results of joint generation models. We visualize the depth estimation results of JointNet, UniCon, and our method on the NYUv2, ScanNet, MSCOCO dataset [17, 38, 56]. Our method shows sharp and fine-detailed depth visualization, which aligns with the t…
Figure 14
Figure 14. Figure 14: Depth-conditioned image generation results of JointNet, UniCon, and Ours. JointNet and UniCon often fail to reflect the text prompt properly, e.g., the green dog generated by UniCon and the flower with green petals generated by JointNet. Our JointDiT generates images …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 54 canonical work pages

  1. [1]

    Depthformer: Multi- scale vision transformer for monocular depth estimation with global local information fusion

    Ashutosh Agarwal and Chetan Arora. Depthformer: Multi- scale vision transformer for monocular depth estimation with global local information fusion. In2022 IEEE international conference on image processing (ICIP), pages 3873–3877. IEEE, 2022. 5, 8

    work page 2022

  2. [2]

    Multimae: Multi-modal multi-task masked autoen- coders

    Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoen- coders. InEuropean Conference on Computer Vision, pages 348–367. Springer, 2022. 5, 6

    work page 2022

  3. [3]

    4m-21: An any-to-any vision model for tens of tasks and modalities.Advances in Neural Infor- mation Processing Systems, 37:61872–61911, 2024

    Roman Bachmann, O ˘guzhan F Kar, David Mizrahi, Ali Gar- jani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin De- hghan, and Amir Zamir. 4m-21: An any-to-any vision model for tens of tasks and modalities.Advances in Neural Infor- mation Processing Systems, 37:61872–61911, 2024. 5, 6

    work page 2024

  4. [4]

    Multidiffusion: Fusing diffusion paths for controlled image generation

    Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. 2023. 4

    work page 2023

  5. [5]

    Se- mantickitti: A dataset for semantic scene understanding of lidar sequences

    Jens Behley, Martin Garbade, Andres Milioto, Jan Quen- zel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Se- mantickitti: A dataset for semantic scene understanding of lidar sequences. InProceedings of the IEEE/CVF inter- national conference on computer vision, pages 9297–9307,

  6. [6]

    Loosec- ontrol: Lifting controlnet for generalized depth conditioning

    Shariq Farooq Bhat, Niloy Mitra, and Peter Wonka. Loosec- ontrol: Lifting controlnet for generalized depth conditioning. InACM SIGGRAPH 2024 Conference Papers, pages 1–11,

    2024

  7. [7]

    Flux.1.https://huggingface

    Black Forest Labs. Flux.1.https://huggingface. co/black-forest-labs/FLUX, 2024. 1-dev. 1, 2, 3, 4, 5

    work page 2024

  8. [8]

    Diffusion forcing: Next-token prediction meets full-sequence diffu- sion.Advances in Neural Information Processing Systems, 37:24081–24125, 2025

    Boyuan Chen, Diego Mart ´ı Mons ´o, Yilun Du, Max Sim- chowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffu- sion.Advances in Neural Information Processing Systems, 37:24081–24125, 2025. 3

    work page 2025

Show all 73 references
  1. [9]

    Text2tex: Text-driven tex- ture synthesis via diffusion models

    Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven tex- ture synthesis via diffusion models. InIEEE International Conference on Computer Vision (ICCV), 2023. 8

    2023

  2. [10]

    Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023. 8

    arXiv

  3. [11]

    Neural ordinary differential equa- tions

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equa- tions. InAdvances in Neural Information Processing Sys- tems (NeurIPS), 2018. 3, 8

    2018

  4. [12]

    Deep diffusion image prior for efficient ood adaptation in 3d inverse problems

    Hyungjin Chung and Jong Chul Ye. Deep diffusion image prior for efficient ood adaptation in 3d inverse problems. In European Conference on Computer Vision, pages 432–455. Springer, 2024. 1

    2024

  5. [13]

    Improving diffusion models for inverse prob- lems using manifold constraints.Advances in Neural Infor- mation Processing Systems, 35:25683–25696, 2022

    Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse prob- lems using manifold constraints.Advances in Neural Infor- mation Processing Systems, 35:25683–25696, 2022

    2022

  6. [14]

    Solving 3d inverse problems us- ing pre-trained 2d diffusion models

    Hyungjin Chung, Dohoon Ryu, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Solving 3d inverse problems us- ing pre-trained 2d diffusion models. ieee. InCVF Conference on Computer Vision and Pattern Recognition, pages 10–12,

  7. [15]

    Latentpaint: Image inpainting in latent space with diffusion models

    Ciprian Corneanu, Raghudeep Gadde, and Aleix M Mar- tinez. Latentpaint: Image inpainting in latent space with diffusion models. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 4334– 4343, 2024. 1

    2024

  8. [16]

    Diffedit: Diffusion-based seman- tic image editing with mask guidance.arXiv preprint arXiv:2210.11427, 2022

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based seman- tic image editing with mask guidance.arXiv preprint arXiv:2210.11427, 2022. 1

    2022 arXiv

  9. [17]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 5, 6, 1, 3, 4, 9

    2017

  10. [18]

    Scaling vision transformers to 22 billion pa- rameters

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdul- mohsin, et al. Scaling vision transformers to 22 billion pa- rameters. InInternational Conference on Machine ...

    2023

  11. [19]

    Scaling rec- tified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rec- tified flow transformers for high-resolution image syn...

  12. [20]

    The pascal visual object classes (voc) challenge.International journal of computer vision, 88:303–338, 2010

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge.International journal of computer vision, 88:303–338, 2010. 6, 1

    2010

  13. [21]

    Geowiz- ard: Unleashing the diffusion priors for 3d geometry esti- mation from a single image

    Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowiz- ard: Unleashing the diffusion priors for 3d geometry esti- mation from a single image. InEuropean Conference on Computer Vision, pages 241–258. Springer, 2024. 2, 5, 6, 8, 1

    2024

  14. [22]

    Fischer, Ulrich Prestel, Pingchuan Ma, Dmytro Kotovenko, Olga Grebenkova, Stefan Andreas Baumann, Vincent Tao Hu, and Bj ¨orn Ommer

    Ming Gui, Johannes S. Fischer, Ulrich Prestel, Pingchuan Ma, Dmytro Kotovenko, Olga Grebenkova, Stefan Andreas Baumann, Vincent Tao Hu, and Bj ¨orn Ommer. Depthfm: Fast monocular depth estimation with flow matching. In AAAI Conference on Artificial Intelligence (AAAI), 2025. 2, 8

    2025

  15. [23]

    Efficient diffu- sion training via min-snr weighting strategy

    Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffu- sion training via min-snr weighting strategy. InProceedings of the IEEE/CVF international conference on computer vi- sion, pages 7441–7451, 2023. 4

    2023

  16. [24]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017. 7, 2

    2017

  17. [25]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022. 5

    2022 arXiv

  18. [26]

    Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 1, 8

    2020

  19. [27]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022. 3, 4, 2

    2022

  20. [28]

    Zero-shot depth completion via test-time align- ment with affine-invariant depth prior

    Lee Hyoseok, Kyeong Seon Kim, Kwon Byung-Ki, and Tae- Hyun Oh. Zero-shot depth completion via test-time align- ment with affine-invariant depth prior. InProceedings of the AAAI Conference on Artificial Intelligence, pages 3877– 3885, 2025. 1, 8

    2025

  21. [29]

    Mixture of diffusers for scene composition and high resolution image generation.arXiv preprint arXiv:2302.02412, 2023

    ´Alvaro Barbero Jim ´enez. Mixture of diffusers for scene composition and high resolution image generation.arXiv preprint arXiv:2302.02412, 2023. 4

    2023 arXiv

  22. [30]

    Dy- namicstereo: Consistent dynamic depth from stereo videos

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Dy- namicstereo: Consistent dynamic depth from stereo videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13229–13239, 2023. 6, 2

    2023

  23. [31]

    Imagic: Text-based real image editing with diffusion models

    Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6007–6017, 2023. 1

    2023

  24. [32]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9492– ...

    2024

  25. [33]

    Openimages: A public dataset for large-scale multi-label and multi-class im- age classification.Dataset available from https://github

    Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Ui- jlings, Stefan Popov, Andreas Veit, et al. Openimages: A public dataset for large-scale multi-label and multi-class im- age classification.Dataset available fro...

    2017

  26. [34]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 1

    2023

  27. [35]

    A sim- ple approach to unifying diffusion-based conditional gener- ation

    Xirui Li, Charles Herrmann, Kelvin CK Chan, Yinxiao Li, Deqing Sun, Chao Ma, and Ming-Hsuan Yang. A sim- ple approach to unifying diffusion-based conditional gener- ation. InInternational Conference on Learning Representa- tions (ICLR), 2025. 2, 3, 5, 6, 8, 1, 4

    2025

  28. [36]

    Matrixcity: A large-scale city dataset for city-scale neural rendering and beyond

    Yixuan Li, Lihan Jiang, Linning Xu, Yuanbo Xiangli, Zhen- zhi Wang, Dahua Lin, and Bo Dai. Matrixcity: A large-scale city dataset for city-scale neural rendering and beyond. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3205–3215, 2023. 6, 2

    2023

  29. [37]

    Revisiting stereo depth estimation from a sequence- to-sequence perspective with transformers

    Zhaoshuo Li, Xingtong Liu, Nathan Drenkow, Andy Ding, Francis X Creighton, Russell H Taylor, and Mathias Un- berath. Revisiting stereo depth estimation from a sequence- to-sequence perspective with transformers. InProceedings of the IEEE/CVF international conference on compute...

    2021

  30. [38]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  31. [39]

    Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022. 3, 8

    2022 arXiv

  32. [40]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 5, 2, 4

    2023

  33. [41]

    Zero-1-to- 3: Zero-shot one image to 3d object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- makov, Sergey Zakharov, and Carl V ondrick. Zero-1-to- 3: Zero-shot one image to 3d object. InProceedings of the IEEE/CVF international conference on computer vision, pages 9298–9309, 2023. 8

    2023

  34. [42]

    Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv preprint arXiv:2206.08916, 2022

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mot- taghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv preprint arXiv:2206.08916, 2022. 5, 6

    2022 arXiv

  35. [43]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. InProceed- ings of the IEEE/CVF Conference on Computer Vision a...

    2024

  36. [44]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022. 1

    2022

  37. [45]

    Readout guidance: Learning con- trol from diffusion features

    Grace Luo, Trevor Darrell, Oliver Wang, Dan B Goldman, and Aleksander Holynski. Readout guidance: Learning con- trol from diffusion features. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8217–8227, 2024. 2, 6, 1

    2024

  38. [46]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF inter- national conference on computer vision, pages 4195–4205,

  39. [47]

    Pexels, royalty-free stock footage website.https: //www.pexels.com

    Pexels. Pexels, royalty-free stock footage website.https: //www.pexels.com. Accessed: 2024-09-30. 7

    2024

  40. [48]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  41. [49]

    Ren ´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.IEEE transactions on pattern analysis and machine intelligence, 44(3):1623–1637, 2020. 5, 6, 4

    2020

  42. [50]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. InProceedings of the IEEE/CVF international conference on computer vision, pages 12179–12188, 2021. 3, 5, 6, 8

    2021

  43. [51]

    Hypersim: A photorealistic syn- thetic dataset for holistic indoor scene understanding

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic syn- thetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference o...

    2021

  44. [52]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 5, 8, 2, 3

    2022

  45. [53]

    Imagenet large scale visual recognition challenge.International journal of computer vision, 115:211–252, 2015

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge.International journal of computer vision, 115:211–252, 2015. 7, 3

    2015

  46. [54]

    Improved techniques for training gans.Advances in neural information processing systems, 29, 2016

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans.Advances in neural information processing systems, 29, 2016. 7, 2

    2016

  47. [55]

    A multi-view stereo benchmark with high- resolution images and multi-camera videos

    Thomas Schops, Johannes L Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and An- dreas Geiger. A multi-view stereo benchmark with high- resolution images and multi-camera videos. InProceed- ings of the IEEE conference on computer vision and p...

    2017

  48. [56]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. InComputer Vision–ECCV 2012: 12th Eu- ropean Conference on Computer Vision, Florence, Italy, Oc- tober 7-13, 2012, Proceedings, Part V 12, pages 746–760. ...

    2012

  49. [57]

    Generative modeling by esti- mating gradients of the data distribution.Advances in neural information processing systems, 32, 2019

    Yang Song and Stefano Ermon. Generative modeling by esti- mating gradients of the data distribution.Advances in neural information processing systems, 32, 2019. 8

    2019

  50. [58]

    Improved techniques for training score-based generative models.Advances in neural information processing systems, 33:12438–12448, 2020

    Yang Song and Stefano Ermon. Improved techniques for training score-based generative models.Advances in neural information processing systems, 33:12438–12448, 2020. 8

    2020

  51. [59]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. InInternational Conference on Learning Represen- tations, 2021. 8

    2021

  52. [60]

    Ldm3d: Latent diffusion model for 3d.arXiv preprint arXiv:2305.10853,

    Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao-Yen Tseng, Fabio Nonato, Matthias Muller, et al. Ldm3d: Latent diffusion model for 3d.arXiv preprint arXiv:2305.10853,

    arXiv

  53. [61]

    Sound2vision: Generating diverse visuals from au- dio through cross-modal latent alignment.arXiv preprint arXiv:2412.06209, 2024

    Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, and Tae-Hyun Oh. Sound2vision: Generating diverse visuals from au- dio through cross-modal latent alignment.arXiv preprint arXiv:2412.06209, 2024. 8

    2024 arXiv

  54. [62]

    Soundbrush: Sound as a brush for visual scene editing

    Kim Sung-Bin, Kim Jun-Seong, Junseok Ko, Yewon Kim, and Tae-Hyun Oh. Soundbrush: Sound as a brush for visual scene editing. InProceedings of the AAAI Conference on Artificial Intelligence, pages 7167–7175, 2025. 8

    2025

  55. [63]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921–1930, 2023. 3, 7, 1

    1921

  56. [64]

    Diode: A dense indoor and outdoor depth dataset.arXiv preprint arXiv:1908.00463, 2019

    Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Daniele, Moham- madreza Mostajabi, Steven Basart, Matthew R Walter, et al. Diode: A dense indoor and outdoor depth dataset.arXiv preprint arXiv:1908.00463, 2019. 5, 6, 1

    1908 arXiv

  57. [65]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 8

    2017

  58. [66]

    Irs: A large naturalistic indoor robotics stereo dataset to train deep models for dis- parity and surface normal estimation

    Qiang Wang, Shizhen Zheng, Qingsong Yan, Fei Deng, Kaiyong Zhao, and Xiaowen Chu. Irs: A large naturalistic indoor robotics stereo dataset to train deep models for dis- parity and surface normal estimation. In2021 IEEE Interna- tional Conference on Multimedia and Expo (ICME), ...

    2021

  59. [67]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation.Advances in Neural Information Pro- cessing Systems, 36:15903–15935, 2023

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation.Advances in Neural Information Pro- cessing Systems, 36:15903–15935, 2023. 7, 8, 3, 4

    2023

  60. [68]

    Depth any- thing v2.Advances in Neural Information Processing Sys- tems, 37:21875–21911, 2025

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2.Advances in Neural Information Processing Sys- tems, 37:21875–21911, 2025. 5, 6, 1, 2, 3, 4

    2025

  61. [69]

    Paint- it: Text-to-texture synthesis via deep convolutional texture map optimization and physically-based rendering

    Kim Youwang, Tae-Hyun Oh, and Gerard Pons-Moll. Paint- it: Text-to-texture synthesis via deep convolutional texture map optimization and physically-based rendering. InPro- ceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 4347–4356, 2024. 8

    2024

  62. [70]

    Metta: Single-view to 3d textured mesh reconstruction with test-time adaptation

    Kim Yu-Ji, Hyunwoo Ha, Kim Youwang, Jaeheung Surh, Hyowon Ha, and Tae-Hyun Oh. Metta: Single-view to 3d textured mesh reconstruction with test-time adaptation. In British Machine Vision Conference (BMVC), 2024. 8

    2024

  63. [71]

    Joint- net: Extending text-to-image diffusion for dense distribution modeling

    Jingyang Zhang, Shiwei Li, Yuanxun Lu, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, and Yao Yao. Joint- net: Extending text-to-image diffusion for dense distribution modeling. InInternational Conference on Learning Repre- sentations (ICLR), 2024. 2, 5, 6, 8, 1, 3, 4

    2024

  64. [72]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023. 2, 6, 8, 1

    2023

  65. [73]

    TNBMM CMBDL LJUUFO CBMBODJOH B MFWJUBUJOH QPUJPO CPUUMF GJMMFE XJUI TIJNNFSJOH CMVF MJRVJEu t1BTUB XJUI NVTISPPNT BOE CBDPOu t

    Tianyi Zheng, Cong Geng, Peng-Tao Jiang, Ben Wan, Hao Zhang, Jinwei Chen, Jia Wang, and Bo Li. Non-uniform timestep sampling: Towards faster diffusion model training. InProceedings of the 32nd ACM International Conference on Multimedia, pages 7036–7045, 2024. 4 JointDiT: Enhan...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.