Pith. sign in

REVIEW 2 major objections 59 references

Perceptual information lives across the entire DiT denoising path; fusing it with spatial weights unlocks multi-modal generation from a frozen backbone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 13:44 UTC pith:NIVW6KI4

load-bearing objection Solid engineering result: multi-timestep adaptive fusion on frozen DiTs is the real gain, and the evaluations back it. the 2 major comments →

arxiv 2606.16673 v2 pith:NIVW6KI4 submitted 2026-06-15 cs.CV

MMDiff: Extending Diffusion Transformers for Multi-Modal Generation

Yagmur Akarken , Orest Kupyn , Christian Rupprecht This is my paper
classification cs.CV
keywords diffusion transformersmulti-modal generationmulti-timestep feature fusiondense predictionsynthetic dataconcept attentionfrozen backbonesemantic segmentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Image generators compute thousands of intermediate representations while they denoise, then throw them away once the picture appears. This paper argues that those discarded features already encode the dense perceptual signals people need—segmentation, depth, saliency—and that the information is spread across many timesteps rather than concentrated at the final clean latent. By training only lightweight decoder heads on a frozen diffusion transformer, and by fusing features from several carefully chosen timesteps with per-pixel adaptive weights, the same model can emit an image together with any combination of those labels in one pass. The result is competitive with specialized discriminative encoders, complementary when the two are combined, and strong enough to generate high-quality synthetic training sets without manual annotation.

Core claim

Perceptual information inside a diffusion transformer is temporally distributed along the denoising trajectory; multi-timestep feature fusion with spatially varying aggregation weights is essential and improves semantic segmentation by up to 28.7% mIoU over any single-timestep extraction. Frozen DiT features are competitive with, and complementary to, state-of-the-art discriminative encoders, so lightweight heads alone turn a generative model into a multi-modal labeler.

What carries the argument

Multi-timestep token aggregation: features drawn from four single-stream layers and four denoising timesteps are projected, then combined by a small temporal transformer that predicts per-pixel mixing weights, refined by CBAM and a gated skip from the cleanest timestep. Concept-driven one-way attention supplies task-specific spatial maps without altering generation.

Load-bearing premise

The four hand-picked timesteps and four layers chosen from entropy and correlation plots on one backbone are assumed to capture the bulk of the useful information; if the peaks lie elsewhere, the fusion advantage shrinks.

What would settle it

Re-run the single-versus-multi-timestep ablation on the same frozen backbone but with a denser or differently spaced set of timesteps; if the mIoU gap collapses below a few points, the claim that information is broadly distributed and that the reported fusion is essential is weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single frozen DiT can emit an image plus segmentation, depth, and saliency in one forward pass, removing the need for separate task networks at generation time.
  • Synthetic datasets with pixel-level labels become far cheaper: generate once, decode many modalities, and train downstream models that beat prior U-Net extraction pipelines.
  • Generative and discriminative features can be concatenated for free performance gains without fine-tuning either backbone.
  • The same aggregation pattern transfers to video DiTs for geometry-aware 3D reconstruction from latents.
  • As stronger DiT generators appear, the same frozen heads improve automatically because richer internal features yield better dense predictions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same multi-timestep aggregation works for any dense label that can be supervised with a lightweight head, future multi-modal generators may ship with a standard ‘perception port’ rather than task-specific fine-tunes.
  • The complementarity result suggests generative pre-training and contrastive pre-training are encoding orthogonal axes of visual structure; joint feature banks could become the default for dense prediction.
  • Because the backbone stays frozen, the method is a drop-in annotation engine for any already-deployed DiT, lowering the barrier to large-scale synthetic data for domains where labels are scarce.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. MMDiff freezes a diffusion transformer (primarily FLUX.1-dev) and trains only lightweight decoder heads to jointly emit images plus dense modalities (semantic segmentation, saliency, depth). The central technical claim is that perceptual information is temporally distributed along the DiT denoising trajectory, so multi-timestep feature fusion with spatially varying aggregation weights is essential; Table 5 shows single-stream t=0 at 50.20 mIoU rising to 78.90 with multi-timestep + concept attention (+28.7 points). Features are taken from single-stream layers L={4,15,26,37} and timesteps T={0.78,0.52,0.26,0.001}, projected and fused by a temporal transformer that predicts per-pixel alphas, refined by CBAM and a gated skip from t=0 (Eq. 2). Concept-driven one-directional attention supplies task-specific spatial maps, and optional DINOv3 features are shown to be complementary. Evaluation uses three protocols: multi-modal consistency on generated images (Table 1), synthetic-data transfer to EoMT/IS-Net/DepthFormer (Tables 2–3), and real-image feature quality under identical decoder training (Table 4), plus ablations, limited-data regimes, SD 3.5 transfer, and a video-to-3DGS probe.

Significance. If the multi-timestep result holds, the paper supplies a practical, frozen-backbone route to multi-modal generation and large-scale synthetic annotation that substantially outperforms prior U-Net extractors (DatasetDM, DiffuMask, Dataset Diffusion) while remaining competitive with fine-tuned discriminative encoders. The controlled re-training of baselines under identical decoders and schedules, the explicit isolation of the +28.7 mIoU jump in Table 5, the complementarity result with DINOv3, and the transfer experiments (SD 3.5, limited data, video→3DGS) are concrete strengths that make the work useful beyond a single backbone. The framework is immediately actionable for synthetic dataset creation without task-specific fine-tuning of the generator.

major comments (2)
  1. Section 3.1 / 3.4 and Table 5: the headline +28.7 mIoU gain is measured from single-stream t=0 (50.20) to multi-timestep + concept attention (78.90). The four timesteps and four layers are hand-chosen from FLUX entropy/correlation plots (Fig. 3). While Appendix G shows the same recipe transfers to SD 3.5 without retuning (70.14 mIoU diffusion-only) and the adaptive-alpha ablation (full model w/o aggregation 78.21 vs 84.95) supports learned fusion, a short sensitivity check—e.g., random or uniform timestep subsets of the same cardinality, or a denser schedule—would make the “temporally distributed / multi-timestep is essential” claim less dependent on this particular schedule. This is a load-bearing engineering choice for the central claim and should be tightened before final acceptance.
  2. Section 4.2 / Table 1: multi-modal consistency relies on pseudo-GT from EoMT, Depth Anything v2 and S3OD plus manual correction on each method’s own generated images. The protocol is carefully described and the ranking is consistent with the other two evaluation axes, but residual method-specific bias cannot be fully ruled out. A brief quantification of the manual-correction rate or an inter-annotator check on a subset would strengthen the claim that DiT features “more accurately capture the visual content” than U-Net extractors.

Circularity Check

0 steps flagged

No significant circularity; multi-timestep gains and feature claims rest on controlled ablations and external benchmarks, not definitional or self-referential reductions.

full rationale

The paper's load-bearing claim (perceptual information temporally distributed in DiT; multi-timestep fusion with spatially varying weights essential, +28.7 mIoU) is an empirical ablation result (Table 5: single-stream t=0 at 50.20 mIoU vs. multi-timestep + concept attention at 78.90). Aggregation weights α are learned parameters of a lightweight transformer head trained on real labeled data, then applied to frozen FLUX (or SD 3.5) features; this is ordinary supervised learning, not a fit-then-predict of a quantity forced by construction. Downstream evaluation uses held-out real benchmarks (PASCAL VOC, DUTS, NYU) and transfer to synthetic data generation, with independent baselines (DINO-v3, VPD, DatasetDM, etc.) retrained under identical protocols. Concept attention is adopted from external work [16]; layer/timestep choices are motivated by entropy/correlation analysis (Fig. 3) but not definitional. Pseudo-GT consistency (Table 1) is secondary and manually verified. No equation reduces a claimed prediction to its own input; no uniqueness theorem or ansatz is imported via self-citation as load-bearing; no known result is merely renamed. The derivation chain is self-contained against external data.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central claim rests on standard diffusion and transformer machinery plus a small set of hand-chosen extraction points and architectural hyperparameters. No new physical entities are postulated; the free parameters are ordinary engineering choices that affect the reported numbers.

free parameters (5)
  • Timestep set T = {0.78, 0.52, 0.26, 0.001}
    Four discrete denoising timesteps chosen from entropy analysis; different choices could change the fusion gain.
  • Layer set L = {4, 15, 26, 37}
    Four single-stream layers selected for coverage and low redundancy; sparse sampling is a free design choice.
  • Temporal transformer hidden dim / heads / layers = 768 / 8 / 3
    Aggregation module capacity (768-d, 8 heads, 3 layers) is set by hand.
  • LayerScale init and skip-gate init = 1e-4 (or 1e-6 in appendix) / 0.1
    Stability constants for residual branches and gated skip from t=0.
  • Decoder learning-rate schedule = 1e-4 base, half for decoders, EMA 0.999
    AdamW 1e-4 with differential rates and EMA beta=0.999; affects final head performance.
axioms (4)
  • standard math Standard forward diffusion noising formula for real-image feature extraction
    Eq. (1) is the usual DDPM/score-based noising step; assumed valid for FLUX latents.
  • domain assumption Frozen FLUX.1-dev dual-stream DiT produces usable dense features without fine-tuning
    Core premise of the framework; all claims of frozen multi-modal generation rest on it.
  • domain assumption Concept tokens updated by one-directional attention do not alter image generation
    Section 3.2; required for task-specific spatial maps at inference without changing appearance.
  • ad hoc to paper Pseudo-GT from SOTA models (EoMT, Depth Anything v2, S3OD) plus manual correction is a valid consistency benchmark
    Table 1 protocol; necessary because generated images differ across methods.
invented entities (2)
  • Token Aggregation Module (temporal transformer + spatially varying alpha + CBAM + gated skip) no independent evidence
    purpose: Fuse multi-timestep DiT features with content-adaptive weights
    New architectural block introduced for this paper; no independent external evidence beyond the reported ablations.
  • Concept-driven one-directional attention stream for DiT no independent evidence
    purpose: Provide interpretable task-specific spatial guidance without affecting generation
    Adopted/adapted from ConceptAttention; the DiT-specific one-way design is part of the method contribution.

pith-pipeline@v1.1.0-grok45 · 25326 in / 2941 out tokens · 23301 ms · 2026-07-12T13:44:25.435695+00:00 · methodology

0 comments
read the original abstract

Diffusion transformers have demonstrated remarkable generative capabilities, yet the rich perceptual representations computed across their denoising trajectory are discarded once the content is rendered. We present MMDiff, a framework that transforms a frozen diffusion transformer into a multi-modal generative system that jointly produces images alongside any combination of dense perceptual modalities using lightweight decoder heads. Our central finding is that perceptual information is temporally distributed along the denoising trajectory, and that multi-timestep feature fusion with spatially varying aggregation weights is essential, improving semantic segmentation results by up to 28.7% mIoU over single-timestep extraction. We further adopt concept-driven attention extraction for interpretable spatial guidance, and show that frozen diffusion features are competitive with and complementary to state-of-the-art encoders such as DINOv3. By training only lightweight decoder heads on a frozen backbone, we achieve strong performance in semantic segmentation, salient object detection, and depth estimation, and demonstrate that this framework enables effective synthetic data generation at scale.

Figures

Figures reproduced from arXiv: 2606.16673 by Christian Rupprecht, Orest Kupyn, Yagmur Akarken.

Figure 1
Figure 1. Figure 1: MMDiff. We present a method to decode multiple modalities from a frozen Diffusion Transformer. We show that information in DiT is distributed across the de￾noising trajectory, and introduce multi-timestep feature fusion to capture the complete perceptual information encoded across them. Lightweight decoder heads predict seman￾tic segmentation, saliency, and depth with no modifications to the generative mod… view at source ↗
Figure 2
Figure 2. Figure 2: MMDiff Pipeline: Given a text prompt, the diffusion transformer generates an image through its dual-stream and single-stream blocks. Features are extracted, pruned, and combined with concept attention maps and discriminative features, then decoded to produce task-specific predictions for the generated image. For images, DatasetDM [48] trains perception decoders on Stable Diffusion fea￾tures to generate seg… view at source ↗
Figure 3
Figure 3. Figure 3: Analysis of DiT features. (a) Activation magnitudes grow monotonically across layers with high inter-layer correlation, motivating sparse layer sampling. (b) Feature entropy varies substantially across timesteps, capturing complementary coarse￾to-fine information that justifies learned aggregation. capture coarse layouts with high entropy, middle timesteps refine object bound￾aries, and late timesteps pres… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparisons on PASCAL (top), DUTS (middle), and [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Wan-2.1 layer/timestep analysis. Feature behavior across video DiT layers and timesteps mirrors the image-model trends. C Unified Multi-Task Decoder The experiments above train a separate decoder per task. Here we test whether a single decoder can serve multiple tasks from the same frozen MMDiff features. We train unified decoders on VOC+DUTS, VOC+NYU, and all three tasks jointly, with per-task loss weight… view at source ↗
Figure 7
Figure 7. Figure 7: Image L=4 L=37 Flux Feature Maps T=0.52 T=0 Dino Maps L=4 L=37 L=2 L=11 Mask [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: Multi-modal feature extraction for salient object detection. [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Concept attention visualization. Visualization of learned concept-specific attention maps for salient object detection. Each concept attends to distinct seman￾tic aspects: object (salient regions), background (surrounding context), contour (object boundaries), edges (fine-grained edge information), and salient (integrated saliency map). The clear separation between concepts demonstrates that our attention … view at source ↗
Figure 8
Figure 8. Figure 8: Token Aggregation Module. Projected multi-timestep features are aggre￾gated using a temporal transformer, followed by skip-projection and CBAM refinement to produce the final fused token representation. a two stage preparation. First, a FeatureNormalizer applies learnable, timestep specific Layer Normalization to address the high variance of diffusion activations. Second, a FeatureProjector uses independen… view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative results from MMDiff across diverse scenes. Each group shows the synthetic input image, predicted binary mask, predicted semantic segmentation, and predicted depth map [PITH_FULL_IMAGE:figures/full_fig_p029_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative comparisons on NYU Depth V2 examples. [PITH_FULL_IMAGE:figures/full_fig_p030_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Qualitative comparisons on DUTS examples. [PITH_FULL_IMAGE:figures/full_fig_p031_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative comparisons on Pascal VOC examples. [PITH_FULL_IMAGE:figures/full_fig_p032_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 9 linked inside Pith

  1. [1]

    In: 2009 IEEE conference on computer vision and pattern recogni- tion

    Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned salient re- gion detection. In: 2009 IEEE conference on computer vision and pattern recogni- tion. pp. 1597–1604. IEEE (2009)

    2009

  2. [2]

    arXiv preprint arXiv:2603.26599 (2026)

    An, Z., Kupyn, O., Uscidda, T., Colaco, A., Ahuja, K., Belongie, S., Gonzalez- Franco, M., Gazulla, M.T.: Vggrpo: Towards world-consistent video generation with 4d latent reward. arXiv preprint arXiv:2603.26599 (2026)

    arXiv 2026

  3. [3]

    In: International Conference on Learning Representations (ICLR) (2026)

    Bahmani, S., Shen, T., Ren, J., Huang, J., Jiang, Y., Turki, H., Tagliasacchi, A., Lindell, D.B., Gojcic, Z., Fidler, S., Ling, H., Gao, J., Ren, X.: Lyra: Generative 3d scene reconstruction via video diffusion model self-distillation. In: International Conference on Learning Representations (ICLR) (2026)

    2026

  4. [4]

    In: The Twelfth International Conference on Learning Representations (2024)

    Chen, J., YU, J., GE, C., Yao, L., Xie, E., Wang, Z., Kwok, J., Luo, P., Lu, H., Li, Z.: Pixart-$\alpha$: Fast training of diffusion transformer for photorealis- tic text-to-image synthesis. In: The Twelfth International Conference on Learning Representations (2024)

    2024

  5. [5]

    arXiv preprint arXiv:1706.05587 (2017)

    Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)

    Pith/arXiv arXiv 2017

  6. [6]

    In: Proceedings of the European conference on computer vision (ECCV)

    Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV). pp. 801–818 (2018)

    2018

  7. [7]

    arXiv preprint (2025)

    Dai, Y., Jiang, F., Wang, C., Xu, M., Qi, Y.: Fantasyworld: Geometry-consistent world modeling via unified video and 3d prediction. arXiv preprint (2025)

    2025

  8. [8]

    Advances in neural information processing systems34, 8780–8794 (2021)

    Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems34, 8780–8794 (2021)

    2021

  9. [9]

    In: International Conference on Learning Representations (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021)

    2021

  10. [10]

    arXiv preprint arXiv:2601.23286 (2026)

    Du, H., Ye, J., Cong, X., Li, R., Ni, J., Agarwal, A., Zhou, Z., Li, Z., Balestriero, R., Wang, Y.: Videogpa: Distilling geometry priors for 3d-consistent video generation. arXiv preprint arXiv:2601.23286 (2026)

    Pith/arXiv arXiv 2026

  11. [11]

    arXiv preprint arXiv:2507.17657 (2025)

    Erel, Y., Dünkel, O., Dabral, R., Golyanik, V., Theobalt, C., Bermano, A.H.: At- tention (as discrete-time markov) chains. arXiv preprint arXiv:2507.17657 (2025)

    arXiv 2025

  12. [12]

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., Rombach, R.:Scalingrectifiedflowtransformersforhigh-resolutionimagesynthesis.In:Forty- first International Conference on Machine Learning (2024)

    2024

  13. [13]

    In: Proceedings of the IEEE international conference on computer vision

    Fan, D.P., Cheng, M.M., Liu, Y., Li, T., Borji, A.: Structure-measure: A new way to evaluate foreground maps. In: Proceedings of the IEEE international conference on computer vision. pp. 4548–4557 (2017)

    2017

  14. [14]

    arXiv preprint arXiv:2505.18584 (2025) 16 Y

    Gan, C., Tu, Y., Chen, X., Chen, T., Li, Y., Harandi, M., Lin, W.: Unleashing diffusiontransformersforvisualcorrespondencebymodulatingmassiveactivations. arXiv preprint arXiv:2505.18584 (2025) 16 Y. Akarken et al

    arXiv 2025

  15. [15]

    Gemma Team: Gemma 3 (2025),https://goo.gle/Gemma3Report

    2025

  16. [16]

    Helbling, A., Meral, T.H.S., Hoover, B., Yanardag, P., Chau, D.H.: Conceptatten- tion: Diffusion transformers learn highly interpretable features (2025)

    2025

  17. [17]

    arXiv preprint arXiv:1606.08415 (2016)

    Hendrycks, D.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)

    Pith/arXiv arXiv 2016

  18. [18]

    Advances in neural information processing systems33, 6840–6851 (2020)

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020)

    2020

  19. [19]

    arXiv preprint (2025)

    Huang, C.H.P., Mitra, N., Jeong, H., Yoon, J.S., Ceylan, D.: Jog3r: Towards 3d- consistent video generators. arXiv preprint (2025)

    2025

  20. [20]

    ACM Transactions on Graphics42(4) (July 2023)

    Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics42(4) (July 2023)

    2023

  21. [21]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Kerssies, T., Cavagnero, N., Hermans, A., Norouzi, N., Averta, G., Leibe, B., Dubbelman, G., de Geus, D.: Your vit is secretly an image segmentation model. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 25303–25313 (2025)

    2025

  22. [22]

    arXiv preprint arXiv:2509.18096 (2025)

    Kim, C., Shin, H., Hong, E., Yoon, H., Arnab, A., Seo, P.H., Hong, S., Kim, S.: Seg4diff: Unveiling open-vocabulary segmentation in text-to-image diffusion transformers. arXiv preprint arXiv:2509.18096 (2025)

    arXiv 2025

  23. [23]

    In: Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision

    Krishnan, A., Yan, X., Casser, V., Kundu, A.: Orchid: Image latent diffusion for joint appearance and geometry generation. In: Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision. pp. 28217–28227 (2025)

    2025

  24. [24]

    arXiv preprint arXiv:2509.22650 (2025)

    Kukleva, A., Simsar, E., Tonioni, A., Naeem, M.F., Tombari, F., Lenssen, J.E., Schiele, B.: Refam: Attention magnets for zero-shot referral segmentation. arXiv preprint arXiv:2509.22650 (2025)

    arXiv 2025

  25. [25]

    In: International Conference on Learning Represen- tations (ICLR) (2026)

    Kupyn, O., Kataoka, H., Rupprecht, C.: S3od: Towards generalizable salient object detection with synthetic data. In: International Conference on Learning Represen- tations (ICLR) (2026)

    2026

  26. [26]

    arXiv preprint arXiv:2510.21615 (2025)

    Kupyn,O.,Manhardt,F.,Tombari,F.,Rupprecht,C.:Epipolargeometryimproves video generation models. arXiv preprint arXiv:2510.21615 (2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Labs, B.F.: Flux.https://github.com/black-forest-labs/flux(2023)

    2023

  28. [28]

    Machine Intelli- gence Research20(6), 837–854 (2023)

    Li, Z., Chen, Z., Liu, X., Jiang, J.: Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation. Machine Intelli- gence Research20(6), 837–854 (2023)

    2023

  29. [29]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Liang, H., Cao, J., Goel, V., Qian, G., Korolev, S., Terzopoulos, D., Plataniotis, K.N., Tulyakov, S., Ren, J.: Wonderland: Navigating 3d scenes from a single image. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 798–810 (2025)

    2025

  30. [30]

    arXiv preprint arXiv:1711.05101 (2017)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    Pith/arXiv arXiv 2017

  31. [31]

    Advances in Neural Information Processing Systems36, 47500–47510 (2023)

    Luo, G., Dunlap, L., Park, D.H., Holynski, A., Darrell, T.: Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems36, 47500–47510 (2023)

    2023

  32. [32]

    Advances in Neural Information Processing Systems36, 76872–76892 (2023)

    Nguyen, Q., Vu, T., Tran, A., Nguyen, K.: Dataset diffusion: Diffusion-based syn- thetic data generation for pixel-level semantic segmentation. Advances in Neural Information Processing Systems36, 76872–76892 (2023)

    2023

  33. [33]

    Peebles,W.,Xie,S.:Scalablediffusionmodelswithtransformers.In:Proceedingsof the IEEE/CVF international conference on computer vision. pp. 4195–4205 (2023)

    2023

  34. [34]

    In: 2012 IEEE conference on computer vision and pattern recognition

    Perazzi, F., Krähenbühl, P., Pritch, Y., Hornung, A.: Saliency filters: Contrast based filtering for salient region detection. In: 2012 IEEE conference on computer vision and pattern recognition. pp. 733–740. IEEE (2012) MMDiff: Extending Diffusion Transformers for Multi-Modal Generation 17

    2012

  35. [35]

    SIAM journal on control and optimization30(4), 838–855 (1992)

    Polyak, B.T., Juditsky, A.B.: Acceleration of stochastic approximation by averag- ing. SIAM journal on control and optimization30(4), 838–855 (1992)

    1992

  36. [36]

    In: European Conference on Computer Vision

    Qin, X., Dai, H., Hu, X., Fan, D.P., Shao, L., Van Gool, L.: Highly accurate di- chotomous image segmentation. In: European Conference on Computer Vision. pp. 38–56. Springer (2022)

    2022

  37. [37]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 12179–12188 (2021)

    2021

  38. [38]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)

    2022

  39. [39]

    In: International Conference on Medical image computing and computer-assisted intervention

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedi- cal image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)

    2015

  40. [40]

    arXiv preprint arXiv:2508.10104 (2025)

    Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khali- dov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al.: Dinov3. arXiv preprint arXiv:2508.10104 (2025)

    Pith/arXiv arXiv 2025

  41. [41]

    In: Interna- tional Conference on Learning Representations (2021)

    Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score- based generative modeling through stochastic differential equations. In: Interna- tional Conference on Learning Representations (2021)

    2021

  42. [42]

    The journal of machine learning research15(1), 1929–1958 (2014)

    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research15(1), 1929–1958 (2014)

    1929

  43. [43]

    Advances in Neural Information Processing Systems 36, 1363–1389 (2023)

    Tang, L., Jia, M., Wang, Q., Phoo, C.P., Hariharan, B.: Emergent correspon- dence from image diffusion. Advances in Neural Information Processing Systems 36, 1363–1389 (2023)

    2023

  44. [44]

    In: Proceedings of the IEEE/CVF international confer- ence on computer vision

    Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., Jégou, H.: Going deeper with image transformers. In: Proceedings of the IEEE/CVF international confer- ence on computer vision. pp. 32–42 (2021)

    2021

  45. [45]

    arXiv preprint arXiv:2503.20314 (2025)

    Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.W., Chen, D., Yu, F., Zhao, H., Yang, J., Zeng, J., Wang, J., Zhang, J., Zhou, J., Wang, J., Chen, J., Zhu, K., Zhao, K., Yan, K., Huang, L., Feng, M., Zhang, N., Li, P., Wu, P., Chu, R., Feng, R., Zhang, S., Sun, S., Fang, T., Wang, T., Gui, T., Weng, T., Shen, T., Lin, W., Wang, W., Wang, W., Zhou, W.,...

    Pith/arXiv arXiv 2025

  46. [46]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B., Revaud, J.: Dust3r: Geometric 3d vision made easy. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 20697–20709 (2024)

    2024

  47. [47]

    In: Proceedings of the European conference on computer vision (ECCV)

    Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19 (2018)

    2018

  48. [48]

    Advances in Neural Information Processing Systems36, 54683–54695 (2023)

    Wu, W., Zhao, Y., Chen, H., Gu, Y., Zhao, R., He, Y., Zhou, H., Shou, M.Z., Shen, C.: Datasetdm: Synthesizing data with perception annotations using diffu- sion models. Advances in Neural Information Processing Systems36, 54683–54695 (2023)

    2023

  49. [49]

    In: 18 Y

    Wu, W., Zhao, Y., Shou, M.Z., Zhou, H., Shen, C.: Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In: 18 Y. Akarken et al. Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1206–1217 (2023)

    2023

  50. [50]

    Xu, G., Ge, Y., Liu, M., Fan, C., Xie, K., Zhao, Z., Chen, H., Shen, C.: What matters when repurposing diffusion models for general dense perception tasks? In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  51. [51]

    Advances in Neural Information Processing Systems37, 21875–21911 (2024)

    Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., Zhao, H.: Depth anything v2. Advances in Neural Information Processing Systems37, 21875–21911 (2024)

    2024

  52. [52]

    arXiv preprint arXiv:2401.11430 (2024)

    Yue, Z., Wang, J., Sun, Q., Ji, L., Chang, E.I., Zhang, H., et al.: Explor- ing diffusion time-steps for unsupervised representation learning. arXiv preprint arXiv:2401.11430 (2024)

    Pith/arXiv arXiv 2024

  53. [53]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Zhang, Q., Zhai, S., Martin, M.A.B., Miao, K., Toshev, A., Susskind, J., Gu, J.: World-consistent video diffusion with explicit 3d modeling. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 21685–21695 (2025)

    2025

  54. [54]

    In: CVPR

    Zhang, Q., Zhai, S., Martin, M.A.B., Miao, K., Toshev, A., Susskind, J., Gu, J.: World-consistent video diffusion with explicit 3d modeling. In: CVPR. pp. 21685– 21695 (2025)

    2025

  55. [55]

    arXiv preprint arXiv:2512.05044 (2025)

    Zhang, Y., Wang, Z., Zheng, W., Zhu, Z., Zhou, J., Lu, J.: Joint 3d geometry reconstruction and motion generation for 4d synthesis from a single image. arXiv preprint arXiv:2512.05044 (2025)

    Pith/arXiv arXiv 2025

  56. [56]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhao, W., Rao, Y., Liu, Z., Liu, B., Zhou, J., Lu, J.: Unleashing text-to-image diffu- sion models for visual perception. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5729–5739 (2023)

    2023

  57. [57]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Zheng, S., Bao, Z., Zhao, R., Hebert, M., Wang, Y.X.: Diff-2-in-1: Bridging genera- tion and dense perception with diffusion models. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  58. [58]

    In: SIGGRAPH (2018) MMDiff: Extending Diffusion Transformers for Multi-Modal Generation 19 A Learning from Limited Data MMDiff is particularly valuable when labeled data is limited

    Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: Learn- ing view synthesis using multiplane images. In: SIGGRAPH (2018) MMDiff: Extending Diffusion Transformers for Multi-Modal Generation 19 A Learning from Limited Data MMDiff is particularly valuable when labeled data is limited. By combining complementary features from mult...

    2018

  59. [59]

    object, background, detail, edges

    Synthetic data generated from limited real supervision (1k, 5k, 10.5k) provides substantial improvements, with the best performance achieved by combining synthetic pre-training and real fine-tuning. Syn. Real ♂planeὫ2/d⌢ve/carὁ5/user Ὠ6 /tvmIoU Decoder trained with 1k real images ✗ ✓ 94.3 45.9 94.7 87.1 58.493.890.8 81.8 79.3 MMDiff✗ 91.746.192.4 83.4 71....

    1938