Pith. sign in

REVIEW 3 major objections 7 minor 93 references

A single video backbone can jointly predict eight dense scene properties—depth, normals, semantics, boundaries, human parts, albedo, shading, materials—from datasets that never share annotations, by distilling per-task diffusion experts int

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:58 UTC pith:XIZKYHVJ

load-bearing objection UniD's latent-distillation scheme is a genuine contribution and the geometry results hold up, but the 'competitive on all eight tasks' claim is not supported by its own Table 5. the 3 major comments →

arxiv 2607.21592 v1 pith:XIZKYHVJ submitted 2026-07-23 cs.CV

Unified Video Dense Prediction from Disjoint Data

Yihong Sun , Seoung Wug Oh , Jiahui Huang , Bharath Hariharan , Joon-Young Lee This is my paper
classification cs.CV
keywords unified dense predictionvideo dense predictiondisjoint datasetsdiffusion priorslatent distillationmulti-task learningtemporal consistencycross-task consistency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that a unified dense-prediction model can be trained on datasets that have no annotation overlap and no pseudo-labeling. UniD first fine-tunes a per-task diffusion specialist on each of eight tasks—depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials—and then trains one video backbone to reconstruct all eight task-specific latents through lightweight projectors. The central bet is that a pretrained diffusion model's visual priors can bridge the domain gaps between disjoint training sources, letting the unified model handle scene-task combinations it never saw during training. The paper reports competitive per-task accuracy against specialists, stronger out-of-distribution generalization than a frozen discriminative backbone, and gains in temporal and cross-task consistency, while acknowledging that segmentation-style classification tasks need an extra projector-only fine-tuning step to recover most of the specialist's accuracy.

Core claim

The paper's central claim is that fragmented scene-understanding datasets can support a unified model without co-annotation or pseudo-labeling. UniD works in two stages: per-task specialists built on a pretrained latent diffusion model are fine-tuned on their own datasets to produce task-specific latent embeddings that generalize beyond the training domain; then a single video U-Net backbone is trained to reconstruct all eight specialist latents simultaneously from an input frame or video, using lightweight per-task latent projectors and a temporal gradient-matching loss. The diffusion prior is the bridge: it supplies appearance-invariant features that let the unified model generalize to sce

What carries the argument

The load-bearing mechanism is latent distillation from per-task diffusion specialists into a shared backbone. Each specialist learns to predict a task latent in the frozen VAE latent space of a Stable Diffusion model; the unified model G_phi is a U-Net with the same initialization and a lightweight DPT-head latent projector Psi^k per task, trained with an L1 latent reconstruction loss plus a temporal gradient matching term that suppresses fast-changing regions. Temporal coherence comes from extended self-attention: every self-attention layer attends to keys and values from past frames stored in a memory bank, so tasks with no video annotations inherit temporal smoothing from tasks that do ha

Load-bearing premise

Latent L1 reconstruction (plus temporal gradient matching) preserves enough of each specialist's output—including the high-frequency boundary and class information that cross-entropy-trained segmentation experts encode—for all eight tasks to remain competitive; the paper's own segmentation results show this premise only partly holds.

What would settle it

Train the same eight-task pipeline but replace latent targets with per-pixel pseudo-labels from the frozen specialists at the same resolution; if the pseudo-label variant clearly beats UniD on segmentation while UniD's latent route does not, then latent reconstruction is the information bottleneck. Equivalently, measure high-frequency energy (e.g., spatial Laplacian norm) of the distilled latents for a cross-entropy segmentation specialist: if the unified latents are substantially flatter than the specialist's latents, the paper's own diagnosis predicts the observed mIoU gap.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single forward pass through a shared U-Net produces all eight predictions, cutting inference cost from K sequential specialist passes to one shared pass plus light projectors (1.8x speedup with no memory bank; over 3.5x with a 16-frame memory bank).
  • Tasks trained only on still images (albedo, shading, boundaries, materials, human parts) inherit temporal consistency from video-centric tasks because the unified backbone is trained with temporal gradient matching and can attend to a memory bank at inference.
  • Depth-derived surface normals and directly predicted normals agree better in the unified model than in the specialist ensemble, indicating that the shared backbone aligns geometric cues across tasks.
  • Adding a ninth task requires only training a new specialist and adding a new latent projector; no existing dataset must be re-annotated or re-pseudo-labeled.
  • For semantic, material, and human-part segmentation, the raw latent-distilled model lags its specialist, and a projector-only fine-tuning step is required to recover most of the gap (e.g., Cityscapes mIoU from 37.8 to 58.4).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the latent-distillation framing is right, the same recipe should extend to other dense tasks whose outputs admit a stable latent code—optical flow, saliency, or scene layout—without architectural change; that is a testable claim the paper does not make.
  • The paper's diagnosis that L1 distillation flattens high-frequency structure (spatial Laplacian norms 2.52 vs. 1.40) predicts that a distillation objective with explicit high-frequency or boundary-preserving terms—or a contrastive or token-level loss for classification specialists—should close the segmentation gap without the fine-tuning stage; comparing such losses on the same eight-task setup wo
  • One practical corollary the authors leave implicit: because temporal consistency transfers across tasks, video annotation for every task may be unnecessary—future benchmarks could measure how much video data for one task (e.g., depth) is enough to stabilize the other seven.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces UniD, a unified video dense prediction model that jointly predicts eight scene properties—depth, surface normals, semantic segmentation, instance boundaries, human parts, albedo, shading, and materials—from disjoint, domain-specific datasets. The method first trains per-task specialists using a single-step latent diffusion model with task-specific pixel projectors, then distills the frozen specialists' latent embeddings into a shared U-Net backbone via lightweight per-task latent projectors, supervised by ℓ1 latent reconstruction and a temporal gradient matching loss. Extended self-attention over a memory bank enables streaming video inference. The paper reports competitive performance on geometric/intrinsic tasks, strong out-of-distribution generalization, and improved temporal and cross-task consistency, while acknowledging in the Limitations that latent reconstruction is less effective for classification tasks and requires an additional projector-only fine-tuning step.

Significance. If the claims held as stated, UniD would be a valuable contribution: it avoids co-annotated data and pseudo-labeling for multi-task dense prediction, leverages diffusion priors to bridge domain gaps, and yields a single efficient video backbone. The manuscript is thorough in scope—eight tasks, diverse benchmarks, OOD evaluations, temporal/cross-task consistency, efficiency, and ablations—and promises code. However, the central claim of 'competitive performance' across all eight tasks is undermined by the segmentation results (Table 5), which are far below per-task specialists and the DINOv3-H baseline even after the extra fine-tuning stage. The paper's real strength lies in geometric and intrinsic tasks, where the unified model is competitive or superior on OOD benchmarks. The significance is therefore real but more narrowly confined than the abstract suggests.

major comments (3)
  1. [Abstract / §4.1, Table 5] The abstract claims 'competitive performance against per-task specialists and multi-task baselines' without qualification, but Table 5 shows large gaps on all three segmentation tasks. Before fine-tuning, UniD (Gϕ) achieves 37.8 mIoU on Cityscapes vs. 64.1 for the specialist and 65.6 for DINOv3-H; 35.7 vs. 47.6 and 62.4 on ADE20K; 40.6 vs. 47.1 and 50.6 on DMS. After the projector-only fine-tuning (UniD (Gϕ)+FT), the model still trails DINOv3-H by ~7 points on Cityscapes (58.4 vs. 65.6) and ~18 points on ADE20K (44.8 vs. 62.4). The Limitations section concedes that 'latent reconstruction is less effective for classification tasks.' This directly contradicts the unqualified 'competitive on all eight tasks' claim. Please either re-scope the central claim to the geometric/intrinsic tasks where the results are competitive, or provide evidence that the fine-tuned model is competitive against
  2. [§4.1, Segmentation and lightweight fine-tuning] The paper attributes the segmentation gap to high spatial Laplacian norms (2.52 vs. 1.40) in CE-trained latents and argues that the rapid recovery from projector-only fine-tuning 'confirms that the backbone Gϕ does learn semantically rich representations.' This inference is not warranted: even after fine-tuning, the model remains below the specialist and the frozen DINOv3-H baseline on most segmentation benchmarks, showing that the ℓ1-distilled latent space does not fully preserve classification information. To support the claim, the paper should provide a direct evaluation of Gϕ features for segmentation (e.g., linear probing, or a control experiment where the fine-tuned projectors are attached to a frozen, untrained backbone). Without this, the conclusion that the unified representation is preserved across classification tasks is an overstatement.
  3. [§3.2, Eq. (6); §4.2, Table 6] Temporal consistency is a central claimed advantage, but the temporal gradient matching loss has two key hyperparameters—the threshold ε and weight λ—set to 2.0 and 5.0 in Appendix B.2 without any sensitivity analysis. Table 9 ablates video training vs. no video training as a whole, but does not isolate ε or λ. Since the mask in Eq. (6) determines which regions receive temporal gradient supervision, the reported mVC improvements in Table 6 could be sensitive to these choices. Please report ablations over ε and λ (at least for depth and semantic segmentation, the tasks with video annotations) to establish robustness of the temporal consistency results.
minor comments (7)
  1. [§4.2, Appendix A] The text describes extended self-attention as 'parameter-free,' but Appendix A introduces a temporal positional embedding that is learned for video-annotated tasks (and zero-initialized otherwise). Please clarify that the module is not entirely parameter-free, or specify that the positional embedding is fixed/zero for tasks without video supervision.
  2. [Eq. (6)] The notation ⊮[·] is non-standard; use 1[·] or an indicator variable. Also specify the norm (presumably the L2 norm) and the axes over which it is computed, as the current definition is ambiguous.
  3. [Tables 2–3] The specialist notation 'UniD (FDθ)' and 'UniD (FNθ)' is inconsistent with the general 'UniD (Fkθ)' used elsewhere. Please define and use a consistent superscript convention (e.g., F^D_θ, F^N_θ).
  4. [Appendix C.1] The material label mapping for ADE20K is described only by examples ('lake→water, curtain→fabric'). Please provide the complete mapping and the resulting number of material classes to make this evaluation reproducible.
  5. [§4.4] The memory estimate '650GB to 78GB' is presented without derivation. Please include a short calculation or reference so readers can verify the claimed memory reduction.
  6. [§4.4, Table 9] The sentence 'This directly ablates the temporal gradient matching (TGM) loss' is imprecise: setting 'Train w/ Video' to ✗ removes all video training signals, including temporal sampling and temporal attention, not just the TGM loss. Please rephrase to describe the actual ablation.
  7. [Abstract / §1] The abstract states 'all learned from disjoint, domain-specific datasets.' While this is true, the segmentation results require an additional fine-tuning stage (Section B.3) that is not mentioned in the abstract. For transparency, please state that the final deployed model for segmentation includes this fine-tuning, or integrate it into the method description.

Circularity Check

0 steps flagged

No significant circularity: UniD's claims rest on an empirical distillation pipeline evaluated against external benchmarks, with only a non-load-bearing self-citation.

full rationale

The derivation chain is empirical and self-contained against external benchmarks. Task specialists are trained with task-specific losses on disjoint datasets (Eq. 3); the unified backbone is trained to reconstruct the resulting specialist latents (Eq. 5); predictions are decoded through the frozen specialist decoders and pixel projectors (Eq. 7). No equation defines the predicted quantity in terms of itself, and no fitted parameter is relabeled as a prediction. The central claims are tested on held-out datasets (NYUv2, ADE20K, IIW, etc.) and against a same-data DINOv3-H baseline, so the OOD and cross-task claims have independent empirical content. The only author self-citation is [67] for the intuition that adjacent frames share rich correspondences; however, the paper itself ablates the temporal mechanism in Table 9, so the cited prior work is not load-bearing. The acknowledged limitation that latent reconstruction is less effective for classification tasks (Section 4.1 and Limitations) weakens the broad 'competitive on all eight tasks' claim and is a correctness/scope concern, not a circularity. Overall, no derivation reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claim rests on a handful of unproved domain assumptions about the diffusion prior and latent distillation. The most load-bearing is the assumption that L1 latent reconstruction preserves high-frequency classification structure, which the paper itself shows to be false and patches with a fine-tuning step. No new physical or theoretical entities are introduced.

free parameters (6)
  • TGM weight λ = 5.0
    Balances per-frame ℓ1 latent reconstruction against temporal gradient matching in L_rec; hand-set, directly controls temporal consistency.
  • TGM mask threshold ε = 2.0
    Suppresses temporal regularization at fast-changing regions (Eq. 6); chosen by hand and affects which gradients are backpropagated.
  • Boundary negative-class loss weight = 20.0
    Upweights sparse non-boundary pixels in specialist boundary training (Appendix B.1); hand-set to address class imbalance.
  • Far-plane regularization weight = 3.0
    Applied to invalid depth and normal pixels (Appendix B.1); hand-set, affects geometric predictions.
  • Semantic segmentation TGM weight = 3.0
    Added to CE loss for video logits in semantic specialist training (Appendix B.1); hand-set.
  • Latent projector fusion dimension = 256
    DPT-head fusion dimension for each Ψk (Section 3.3); architecture choice.
axioms (5)
  • domain assumption Pretrained Stable Diffusion U-Net and VAE provide a latent space with enough visual priors to bridge disjoint domains and support pixel-decoding of arbitrary dense tasks (Eq. 1).
    The paper's key insight; not proven, only evidenced by OOD results. Section 3.1.
  • domain assumption Specialist latents F_k^θ(E(x)) computed on unlabeled/unseen frames are reliable distillation targets (Eq. 5).
    The unified model is trained entirely against specialist outputs; if these are noisy or domain-biased, the student inherits the bias. Section 3.2.
  • domain assumption L1 loss in latent space plus temporal gradient matching preserves all task-relevant information, including classification logits.
    Paper shows this is false for CE tasks, requiring extra fine-tuning. Section 4.1.
  • domain assumption The frozen VAE decoder D' can decode shared latent embeddings into per-task pixel predictions with only a 3×3 projector.
    Specialists and unified model rely on D' being a fixed, invertible-enough decoder. Section 3.1.
  • domain assumption Extended self-attention over historical frames gives temporal consistency without training on videos for tasks lacking video annotations.
    Ablation supports this, but relies on correspondence from diffusion prior. Eq. 2.

pith-pipeline@v1.3.0-alltime-deepseek · 21586 in / 13221 out tokens · 121959 ms · 2026-08-01T06:58:25.193929+00:00 · methodology

0 comments
read the original abstract

Scene understanding requires simultaneous prediction about geometry, appearance, and semantics. However, existing task-specific annotations are fragmented across incompatible, domain-specific datasets. Current unified systems circumvent this by restricting training to fully co-annotated data, or by incurring the large computational cost of pseudo-labeling. To mitigate this, we introduce UniD, a unified video model that jointly predicts eight dense scene properties-depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials-all learned from disjoint, domain-specific datasets. We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labeling. Our key insight is that the strong visual priors of a pretrained diffusion model are sufficient to bridge the domain gaps introduced by disjoint training sources, enabling robust generalization to scene-task combinations never seen during training. UniD achieves competitive performance against per-task specialists and multi-task baselines, with strong generalization to out-of-distribution scenarios and enhanced temporal and cross-task consistency. Code and video results are available at https://unid-video.github.io/.

Figures

Figures reproduced from arXiv: 2607.21592 by Bharath Hariharan, Jiahui Huang, Joon-Young Lee, Seoung Wug Oh, Yihong Sun.

Figure 1
Figure 1. Figure 1: Unified Video Dense Predictions by UniD. environments with limited semantic diversity, e.g. architectural landscapes. Se￾mantic annotations require manual labeling over a diverse set of scenes, yielding rich category diversity but rarely paired with geometric ground truth. Finally, intrinsic decompositions [5,53] are prohibitively expensive to measure in the real world and largely confined to synthetic dom… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of UniD. (a) Task specialist training. For each task k, a specialist backbone F k θ and pixel projector P k are finetuned on their task-specific dataset Dk with loss L k . Note that both the encoder E and decoder D ′ (in gray) are frozen during training. (b) Unified model training. The unified backbone Gϕ is then trained along with K latent projectors {Ψ k } K k=1 to reconstruct the latents {l k i… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative Results of UniD compared with DINOv3-H.1 4.1 Per-Task Performance Out-of-distribution generalization. The main claim of UniD is that the generative priors of a pretrained diffusion backbone are sufficient to bridge the domain gaps introduced by disjoint training sources, enabling generalization to scene￾task combinations never seen with annotations. To evaluate this, in-distribution benchmarks … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

93 extracted references · 19 linked inside Pith

  1. [1]

    Advances in Neural Information Processing Systems37, 61872–61911 (2024)

    Bachmann, R., Kar, O.F., Mizrahi, D., Garjani, A., Gao, M., Griffiths, D., Hu, J., Dehghan, A., Zamir, A.: 4m-21: An any-to-any vision model for tens of tasks and modalities. Advances in Neural Information Processing Systems37, 61872–61911 (2024)

    2024

  2. [2]

    In: European conference on computer vision

    Bachmann, R., Mizrahi, D., Atanov, A., Zamir, A.: Multimae: Multi-modal multi- task masked autoencoders. In: European conference on computer vision. pp. 348–

  3. [3]

    arXiv preprint arXiv:2502.13078 (2025)

    Badki, A., Su, H., Wen, B., Gallo, O.: L4p: Towards unified low-level 4d vision perception. arXiv preprint arXiv:2502.13078 (2025)

    arXiv 2025

  4. [4]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Bae, G., Davison, A.J.: Rethinking inductive biases for surface normal estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9535–9545 (2024)

    2024

  5. [5]

    IEEE transactions on pattern analysis and machine intelligence37(8), 1670–1687 (2014)

    Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. IEEE transactions on pattern analysis and machine intelligence37(8), 1670–1687 (2014)

    2014

  6. [6]

    Bell,S.,Bala,K.,Snavely,N.:Intrinsicimagesinthewild.ACMTrans.onGraphics (SIGGRAPH)33(4) (2014)

    2014

  7. [7]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Bin, Y., Hu, W., Wang, H., Chen, X., Wang, B.: Normalcrafter: Learning tem- porally consistent normals from video diffusion priors. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8330–8339 (2025)

    2025

  8. [8]

    arXiv preprint arXiv:2311.15127 (2023)

    Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al.: Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 (2023)

    Pith/arXiv arXiv 2023

  9. [9]

    arXiv preprint arXiv:2410.02073 (2024)

    Bochkovskii, A., Delaunoy, A., Germain, H., Santos, M., Zhou, Y., Richter, S.R., Koltun, V.: Depth pro: Sharp monocular metric depth in less than a second. arXiv preprint arXiv:2410.02073 (2024)

    Pith/arXiv arXiv 2024

  10. [10]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 18392–18402 (2023)

    2023

  11. [11]

    In: European conference on computer vision

    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: European conference on computer vision. pp. 611–

  12. [12]

    arXiv preprint arXiv:2001.10773 (2020)

    Cabon, Y., Murray, N., Humenberger, M.: Virtual kitti 2. arXiv preprint arXiv:2001.10773 (2020)

    Pith/arXiv arXiv 2001

  13. [13]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2026)

    Cao, A.Q., Lopes, I., de Charette, R.: Stablemtl: Repurposing latent diffusion mod- els for multi-task learning from partially annotated synthetic datasets. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2026)

    2026

  14. [14]

    ACM Transactions on Graphics (TOG)43(6), 1–12 (2024)

    Careaga, C., Aksoy, Y.: Colorful diffuse intrinsic image decomposition in the wild. ACM Transactions on Graphics (TOG)43(6), 1–12 (2024)

    2024

  15. [15]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Chen, S., Guo, H., Zhu, S., Zhang, F., Huang, Z., Feng, J., Kang, B.: Video depth anything: Consistent depth estimation for super-long videos. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 22831–22840 (2025)

    2025

  16. [16]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, W., Qian, S., Fan, D., Kojima, N., Hamilton, M., Deng, J.: Oasis: A large- scale dataset for single image 3d in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 679–688 (2020)

    2020

  17. [17]

    In: Unified Video Dense Prediction from Disjoint Data 17 Proceedings of the IEEE conference on computer vision and pattern recognition

    Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., Yuille, A.: Detect what you can: Detecting and representing objects using holistic models and body parts. In: Unified Video Dense Prediction from Disjoint Data 17 Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1971–1978 (2014)

    1971

  18. [18]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022)

    2022

  19. [19]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213–3223 (2016)

    2016

  20. [20]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Dai,A.,Chang,A.X.,Savva,M.,Halber,M.,Funkhouser,T.,Nießner,M.:Scannet: Richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5828–5839 (2017)

    2017

  21. [21]

    In: Forty-first international conference on machine learning (2024)

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al.: Scaling rectified flow transformers for high-resolution image synthesis. In: Forty-first international conference on machine learning (2024)

    2024

  22. [22]

    In: European Conference on Computer Vision

    Fu, X., Yin, W., Hu, M., Wang, K., Ma, Y., Tan, P., Shen, S., Lin, D., Long, X.: Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In: European Conference on Computer Vision. pp. 241–258. Springer (2024)

    2024

  23. [23]

    In: 2012 IEEE conference on computer vision and pattern recognition

    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: 2012 IEEE conference on computer vision and pattern recognition. pp. 3354–3361. IEEE (2012)

    2012

  24. [24]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Güler, R.A., Neverova, N., Kokkinos, I.: Densepose: Dense human pose estimation in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7297–7306 (2018)

    2018

  25. [25]

    arXiv preprint arXiv:2409.18124 (2024)

    He, J., Li, H., Yin, W., Liang, Y., Li, L., Zhou, K., Zhang, H., Liu, B., Chen, Y.C.: Lotus: Diffusion-based visual foundation model for high-quality dense prediction. arXiv preprint arXiv:2409.18124 (2024)

    Pith/arXiv arXiv 2024

  26. [26]

    arXiv preprint arXiv:2210.02303 (2022)

    Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 (2022)

    Pith/arXiv arXiv 2022

  27. [27]

    Advances in neural information processing systems33, 6840–6851 (2020)

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020)

    2020

  28. [28]

    arXiv preprint arXiv:2207.12598 (2022)

    Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)

    Pith/arXiv arXiv 2022

  29. [29]

    Advances in neural information processing systems35, 8633– 8646 (2022)

    Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, D.J.: Video diffusion models. Advances in neural information processing systems35, 8633– 8646 (2022)

    2022

  30. [30]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Huang, T.E., Liu, Y., Van Gool, L., Yu, F.: Video task decathlon: Unifying im- age and video tasks in autonomous driving. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8647–8657 (2023)

    2023

  31. [31]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Jain, S., Wang, X., Gonzalez, J.E.: Accel: A corrective fusion network for efficient semantic segmentation on video. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8866–8875 (2019)

    2019

  32. [32]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Kar, O.F., Yeo, T., Atanov, A., Zamir, A.: 3d common corruptions and data aug- mentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18963–18974 (2022) 18 Sun et al

    2022

  33. [33]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., Irani, M.: Imagic: Text-based real image editing with diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6007–6017 (2023)

    2023

  34. [34]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition

    Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R.C., Schindler, K.: Re- purposing diffusion-based image generators for monocular depth estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 9492–9502 (2024)

    2024

  35. [35]

    In: European Conference on Computer Vision

    Kim, D., Cho, S., Kim, S., Luo, C., Hong, S.: Chameleon: A data-efficient generalist for dense visual prediction in the wild. In: European Conference on Computer Vision. pp. 422–441. Springer (2024)

    2024

  36. [36]

    Kim, D., Kim, J., Cho, S., Luo, C., Hong, S.: Universal few-shot learning of densepredictiontaskswithvisualtokenmatching.arXivpreprintarXiv:2303.14969 (2023)

    Pith/arXiv arXiv 2023

  37. [37]

    In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops

    Koch, T., Liebel, L., Fraundorfer, F., Korner, M.: Evaluation of cnn-based single- image depth estimation methods. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops. pp. 0–0 (2018)

    2018

  38. [38]

    arXiv preprint arXiv:2312.14125 (2023)

    Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang, J., Schindler, G., Hornung, R., Birodkar, V., Yan, J., Chiu, M.C., et al.: Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125 (2023)

    Pith/arXiv arXiv 2023

  39. [39]

    Com- puter Vision and Pattern Recognition (CVPR) (2017)

    Kovacs, B., Bell, S., Snavely, N., Bala, K.: Shading annotations in the wild. Com- puter Vision and Pattern Recognition (CVPR) (2017)

    2017

  40. [40]

    Lambert, J., Liu, Z., Sener, O., Hays, J., Koltun, V.: Mseg: A composite dataset for multi-domainsemanticsegmentation.In:ProceedingsoftheIEEE/CVFconference on computer vision and pattern recognition. pp. 2879–2888 (2020)

    2020

  41. [41]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, W.H., Liu, X., Bilen, H.: Learning multiple dense prediction tasks from partially annotated data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18879–18889 (2022)

    2022

  42. [42]

    arXiv preprint arXiv:1809.00716 (2018)

    Li, W., Saeedi, S., McCormac, J., Clark, R., Tzoumanikas, D., Ye, Q., Huang, Y., Tang, R., Leutenegger, S.: Interiornet: Mega-scale multi-sensor photo-realistic indoor scenes dataset. arXiv preprint arXiv:1809.00716 (2018)

    Pith/arXiv arXiv 2018

  43. [43]

    arXiv preprint arXiv:2405.15757 (2024)

    Liang,F.,Kodaira,A.,Xu,C.,Tomizuka,M.,Keutzer,K.,Marculescu,D.:Looking backward: Streaming video-to-video translation with feature banks. arXiv preprint arXiv:2405.15757 (2024)

    Pith/arXiv arXiv 2024

  44. [44]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    Lin, B., Jiang, W., Chen, P., Liu, S., Chen, Y.C.: Mtmamba++: Enhancing multi- task dense scene understanding via mamba-based decoders. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025)

    2025

  45. [45]

    In: European conference on computer vision

    Lin, B., Jiang, W., Chen, P., Zhang, Y., Liu, S., Chen, Y.C.: Mtmamba: Enhancing multi-task dense scene understanding by mamba-based decoders. In: European conference on computer vision. pp. 314–330. Springer (2024)

    2024

  46. [46]

    In: European conference on computer vision

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)

    2014

  47. [47]

    arXiv preprint arXiv:1711.05101 (2017)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    Pith/arXiv arXiv 2017

  48. [48]

    IEEE Transactions on Multimedia26, 6375–6385 (2024)

    Lu, Y., Sirejiding, S., Ding, Y., Wang, C., Lu, H.: Prompt guided transformer for multi-task dense prediction. IEEE Transactions on Multimedia26, 6375–6385 (2024)

    2024

  49. [49]

    Unified Video Dense Prediction from Disjoint Data 19 In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Mehl, L., Schmalfuss, J., Jahedi, A., Nalivayko, Y., Bruhn, A.: Spring: A high- resolutionhigh-detaildatasetandbenchmarkforsceneflow,opticalflowandstereo. Unified Video Dense Prediction from Disjoint Data 19 In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4981–4991 (2023)

    2023

  50. [50]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Miao, J., Wang, X., Wu, Y., Li, W., Zhang, X., Wei, Y., Yang, Y.: Large-scale video panoptic segmentation in the wild: A benchmark. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21033– 21043 (2022)

    2022

  51. [51]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Miao, J., Wei, Y., Wu, Y., Liang, C., Li, G., Yang, Y.: Vspw: A large-scale dataset for video scene parsing in the wild. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4133–4143 (2021)

    2021

  52. [52]

    In: Proceedings of the AAAI conference on artificial intelligence

    Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., Shan, Y.: T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In: Proceedings of the AAAI conference on artificial intelligence. vol. 38, pp. 4296–4304 (2024)

    2024

  53. [53]

    In: Proceedings of the IEEE international conference on computer vision (2015)

    Narihira, T., Maire, M., Yu, S.X.: Direct intrinsics: Learning albedo-shading de- composition by convolutional regression. In: Proceedings of the IEEE international conference on computer vision (2015)

    2015

  54. [54]

    In: Proceedings of the IEEE international conference on computer vision

    Neuhold, G., Ollmann, T., Rota Bulo, S., Kontschieder, P.: The mapillary vistas dataset for semantic understanding of street scenes. In: Proceedings of the IEEE international conference on computer vision. pp. 4990–4999 (2017)

    2017

  55. [55]

    arXiv preprint arXiv:2112.10741 (2021)

    Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)

    Pith/arXiv arXiv 2021

  56. [56]

    https://www.pexels.com/

    Pexels Website: Pexels — free stock photos & videos you can use everywhere. https://www.pexels.com/

  57. [57]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 12179–12188 (2021)

    2021

  58. [58]

    arXiv preprint arXiv:2408.00714 (2024)

    Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)

    Pith/arXiv arXiv 2024

  59. [59]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Roberts, M., Ramapuram, J., Ranjan, A., Kumar, A., Bautista, M.A., Paczan, N., Webb, R., Susskind, J.M.: Hypersim: A photorealistic synthetic dataset for holis- tic indoor scene understanding. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10912–10922 (2021)

    2021

  60. [60]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)

    2022

  61. [61]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Schops, T., Schonberger, J.L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., Geiger, A.: A multi-view stereo benchmark with high-resolution images and multi-camera videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3260–3269 (2017)

    2017

  62. [62]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Shao, J., Yang, Y., Zhou, H., Zhang, Y., Shen, Y., Guizilini, V., Wang, Y., Poggi, M., Liao, Y.: Learning temporally consistent video depth from video diffusion pri- ors. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 22841–22852 (2025)

    2025

  63. [63]

    Shin, I., Kim, D., Yu, Q., Xie, J., Kim, H.S., Green, B., Kweon, I.S., Yoon, K.J., Chen, L.C.: Video-kmax: A simple unified approach for online and near-online videopanopticsegmentation.In:ProceedingsoftheIEEE/CVFWinterConference on Applications of Computer Vision. pp. 229–239 (2024) 20 Sun et al

    2024

  64. [64]

    In: European conference on computer vision

    Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: European conference on computer vision. pp. 746–

  65. [65]

    arXiv preprint arXiv:2508.10104 (2025)

    Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khali- dov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al.: Dinov3. arXiv preprint arXiv:2508.10104 (2025)

    Pith/arXiv arXiv 2025

  66. [66]

    arXiv preprint arXiv:2011.13456 (2020)

    Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score- based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 (2020)

    Pith/arXiv arXiv 2011

  67. [67]

    Advances in neural information processing systems36, 1363– 1389 (2023)

    Tang, L., Jia, M., Wang, Q., Phoo, C.P., Hariharan, B.: Emergent correspondence from image diffusion. Advances in neural information processing systems36, 1363– 1389 (2023)

    2023

  68. [68]

    In: European conference on computer vision

    Upchurch, P., Niu, R.: A dense material segmentation dataset for indoor and out- door scene parsing. In: European conference on computer vision. pp. 450–466. Springer (2022)

    2022

  69. [69]

    arXiv preprint arXiv:1908.00463 (2019)

    Vasiljevic, I., Kolkin, N., Zhang, S., Luo, R., Wang, H., Dai, F.Z., Daniele, A.F., Mostajabi, M., Basart, S., Walter, M.R., et al.: Diode: A dense indoor and outdoor depth dataset. arXiv preprint arXiv:1908.00463 (2019)

    Pith/arXiv arXiv 1908

  70. [70]

    In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    Wang, W., Zhu, D., Wang, X., Hu, Y., Qiu, Y., Wang, C., Hu, Y., Kapoor, A., Scherer, S.: Tartanair: A dataset to push the limits of visual slam. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 4909–4916. IEEE (2020)

    2020

  71. [71]

    arXiv preprint arXiv:2506.20279 (2025)

    Xia, C., Jia, C., Dang, Z., Luo, M., Li, Z., Chang, X.: From ideal to real: Uni- fied and data-efficient dense prediction for real-world scenarios. arXiv preprint arXiv:2506.20279 (2025)

    arXiv 2025

  72. [72]

    Xu, G., Ge, Y., Liu, M., Fan, C., Xie, K., Zhao, Z., Chen, H., Shen, C.: What matters when repurposing diffusion models for general dense perception tasks? arXiv preprint arXiv:2403.06090 (2024)

    Pith/arXiv arXiv 2024

  73. [73]

    IEEE Transactions on Circuits and Systems for Video Technology34(2), 1228–1240 (2023)

    Xu, Y., Li, X., Yuan, H., Yang, Y., Zhang, L.: Multi-task learning with multi-query transformer for dense prediction. IEEE Transactions on Circuits and Systems for Video Technology34(2), 1228–1240 (2023)

    2023

  74. [74]

    In: Proceedings of the AAAI conference on artificial intelligence

    Xu, Y., Yang, Y., Zhang, L.: Demt: Deformable mixer transformer for multi-task learning of dense prediction. In: Proceedings of the AAAI conference on artificial intelligence. vol. 37, pp. 3072–3080 (2023)

    2023

  75. [75]

    arXiv preprint arXiv:2410.10815 (2024)

    Yang, H., Huang, D., Yin, W., Shen, C., Liu, H., He, X., Lin, B., Ouyang, W., He, T.: Depth any video with scalable synthetic data. arXiv preprint arXiv:2410.10815 (2024)

    Pith/arXiv arXiv 2024

  76. [76]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Yang, L., Qi, L., Li, X., Li, S., Jampani, V., Yang, M.H.: Unified dense prediction of video diffusion. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 28963–28973 (2025)

    2025

  77. [77]

    Advances in Neural Information Processing Systems37, 21875–21911 (2024)

    Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., Zhao, H.: Depth anything v2. Advances in Neural Information Processing Systems37, 21875–21911 (2024)

    2024

  78. [78]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Yang, L., Fan, Y., Xu, N.: Video instance segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5188–5197 (2019)

    2019

  79. [79]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Yang, Y., Jiang, P.T., Hou, Q., Zhang, H., Chen, J., Li, B.: Multi-task dense prediction via mixture of low-rank experts. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 27927–27937 (2024)

    2024

  80. [80]

    In: The Thirteenth International Conference on Learning Representations (2025) Unified Video Dense Prediction from Disjoint Data 21

    Yang, Y., Jiang, P.T., Hou, Q., Zhang, H., Chen, J., Li, B.: Multi-task dense predictions via unleashing the power of diffusion. In: The Thirteenth International Conference on Learning Representations (2025) Unified Video Dense Prediction from Disjoint Data 21

    2025

Showing first 80 references.