Pith. sign in

REVIEW 5 major objections 6 minor 84 references

LaVin-DiT: Large Vision Diffusion Transformer

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a single diffusion transformer, LaVin-DiT, can unify more than 20 image and video tasks and outperform the strongest autoregressive large vision model baseline.

desk verdict A genuinely interesting unified vision diffusion transformer whose headline claims are undercut by missing video numbers, an 'unseen tasks' contradiction, and unresolved pseudo-label provenance. read the letter →

arxiv 2411.11505 v4 pith:PLWW3LKE submitted 2024-11-18 cs.CV

Zhaoqing Wang , Xiaobo Xia , Runnan Chen , Dongdong Yu , Changhu Wang , Mingming Gong , Tongliang Liu This is my paper · ORCID
classification cs.CV
keywords diffusiontransformerlargevisionmodelin-contextlearningspatial-temporalvariationalautoencoderflowmatchingmulti-task
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LaVin-DiT sets out to show that a diffusion-based transformer, rather than an autoregressive language-model-style architecture, is the right backbone for a unified large vision model. The paper argues that by compressing images and videos into a continuous latent space with a spatial-temporal variational autoencoder and training one joint diffusion transformer with in-context input-target pairs as task definitions, a single 3.4B model can handle more than 20 image and video tasks without fine-tuning. Across benchmarks it reports consistent gains over the strongest autoregressive baseline, including large margins on NYU-v2 depth estimation and surface normals, and 1.7–2.3× faster inference. The payoff, if true, is a scalable vision foundation model that preserves spatial structure and adapts to tasks by example rather than by task-specific heads.

What carries the argument

The load-bearing machinery is the combination of a spatial-temporal variational autoencoder (ST-VAE) and a joint diffusion transformer (J-DiT). ST-VAE compresses both images and videos into a continuous $4\times 8\times 8$ latent representation using causal 3D convolutions, with two stages of spatial-temporal downsampling followed by spatial-only downsampling, which keeps diffusion computation tractable. J-DiT extends MM-DiT by giving the clean condition latents and the noisy target latents separate patch embeddings and separate adaptive RMS normalization, then lets them interact through full-sequence joint attention with 3D rotary position embeddings so the model knows where each token sits in space and time. Training uses the conditional flow-matching loss to regress the velocity field, and inference integrates the learned ODE backward with Euler steps, so the same network can denoise any task's target given the right context.

What would settle it

Check the training-data documentation of Depth-Anything V2 and Stable-Normal (turbo) for NYUv2; if either model was trained on NYUv2 or its derivatives, the Table 2 depth and normal evaluations measure label inheritance rather than in-context generalization. A cleaner experiment is to retrain LaVin-DiT with pseudo labels from a teacher provably never exposed to NYUv2 and re-measure AbsRel and MAE on the same test set.

Watch

Extended reading notes

Core claim

The paper's central claim is that the conditional distribution $p(y|x,s)$ of a vision output $y$ given a query $x$ and a task context $s$ can be modeled as a flow-matching denoising problem in a shared latent space. The task context is a set of input-target pairs that define the task, the query is encoded with the same spatial-temporal variational autoencoder, and the joint diffusion transformer denoises the target latent in parallel across the full sequence. The paper reports that at 3.4B parameters LaVin-DiT outperforms LVM on foreground segmentation (mIoU 67.87–75.80 versus 47.66–51.29 across splits), single-object detection (mIoU 67.85–69.32 versus 48.25–50.08), NYU-v2 depth (AbsRel 6.2 versus 30.2), surface normals (MAE 15.901 versus 23.433), and ImageNet inpainting (FID 1.65 versus 4.05), while running 1.7–2.3× faster at inference. These numbers are the concrete evidence for the thesis that diffusion modeling is a better fit than sequential token prediction for unified vision.

Load-bearing premise

The load-bearing premise is that the pseudo depth and surface-normal labels generated by Depth-Anything V2 and Stable-Normal (turbo) are accurate and have not already been exposed to the NYUv2 evaluation distribution, since those labels supervise training and the paper's headline geometric numbers are measured on NYUv2.

Editorial extensions

If this is right

  • Diffusion-based vision foundation models can match or beat autoregressive models on dense prediction while being faster, so future large vision models need not borrow LLM-style token-by-token decoding.
  • Longer task-context sets monotonically improve downstream accuracy, making the selection of good input-target pairs a practical lever for performance.
  • Scaling LaVin-DiT from 0.1B to 3.4B parameters produces monotone gains in both training loss and downstream metrics, supporting further scaling efforts.
  • A single checkpoint can serve both image and video tasks without task-specific heads or fine-tuning, simplifying deployment of generalist vision models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the depth and normal training labels come from teacher models, LaVin-DiT's NYU-v2 results likely inherit the teachers' biases; a direct test would be to compare the student's errors on out-of-distribution images with the teachers' errors, and the paper does not report such a comparison.
  • If pseudo-label contamination is ruled out, the strong depth and normal numbers would suggest that in-context conditioning can transfer metric knowledge from a teacher to a student; a clean way to check this is to hold out a teacher's label set and see whether the student still reproduces the teacher's distinctive failure modes.
  • A natural extension is automatic context selection: since longer contexts help, picking the most informative input-target pairs could deliver large gains without any architectural change, a direction the paper names as future work.
  • The limitation stated in the paper implies that in-context generalization should be read as generalization within the neighborhood of the training task distribution; tasks far outside that neighborhood would require either more data or a different mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes LaVin-DiT, a unified diffusion-transformer foundation model that handles over 20 image and video tasks via a spatial-temporal VAE (ST-VAE), a joint diffusion transformer (J-DiT) with full-sequence joint attention and 3D RoPE, and in-context conditioning through input-target pairs. Training data combines about 3.2M images and 0.6M videos, with pseudo-labels for depth and normals from Depth-Anything V2 and Stable-Normal (turbo). Quantitative results are reported for foreground segmentation, single object detection, colorization, NYUv2 depth/normal estimation, and ImageNet inpainting, plus qualitative results for many tasks and a latency comparison against LVM. The paper claims state-of-the-art performance and 1.7–2.3× faster inference relative to LVM.

Significance. If the reported results are reproducible and the issues below are resolved, this is a potentially influential contribution: it shows that a diffusion-based generative framework can unify diverse vision tasks and outperform the autoregressive LVM baseline on several benchmarks. The paper provides useful implementation details, including algorithm pseudocode, architecture configuration tables, and a plan to release code and models, which supports reproducibility. The strongest quantitative gains are on foreground segmentation (e.g., 67.87 mIoU vs. 48.94 for LVM on Split 1), single object detection, colorization, and inpainting (FID 1.65 vs. 4.05). However, several load-bearing claims currently lack sufficient evidence, particularly the depth/normal results, the 'unseen task' claim, and all video-task claims.

major comments (5)
  1. [§4.2, Table 1, Table 4] The claim that foreground segmentation and single object detection are 'unseen tasks during our training' (Table 1 note and main text) is contradicted by Table 4, which lists single object detection on COCO/Object365 and instance segmentation on COCO/ADE20K/Cityscapes as training tasks. Even if the Pascal-5i classes are held out, the task families are trained; this overstates the generalization claim and should be reworded with a precise statement of what is unseen (e.g., dataset splits or classes).
  2. [§4.1, Table 4, Table 2] The NYUv2 depth and surface normal results in Table 2 may be inherited from the pseudo-label teachers. Training depth labels are generated with Depth-Anything V2 and normal labels with Stable-Normal (turbo) on ImageNet-1K/COCO/ADE20K/Cityscapes. The paper does not report whether these teachers or their training data include NYUv2 or NYUv2-derived benchmarks, nor does it provide depth/normal evaluation on a dataset outside the teachers' likely training corpora. Without this provenance, the AbsRel 6.2 and MAE 15.901 gains over LVM could reflect teacher-to-student distillation rather than in-context generalization, so the comparison in Table 2 is not decisive.
  3. [§4.2, §4.3, Table 1, Figure 5] The colorization results are internally inconsistent: Table 1 and the text in §4.2 report MSE 0.24 for LaVin-DiT (3.4B), while §4.3 and Figure 5 report MSE 0.273 for the same model. The evaluation subsample sizes also differ from the protocol in §B.2 (1,000 images vs. possibly different for Figure 5). The authors should state one protocol and report all numbers from it, or explain the discrepancy.
  4. [§4.2, §B.1, Figures 3, 23-29] No quantitative results are reported for any video task, despite the abstract and introduction claiming state-of-the-art performance across 'diverse vision tasks' including video. Frame prediction, video depth/normal/optical flow/instance segmentation, and depth/normal-to-video are shown only qualitatively. Since the spatial-temporal VAE and video evaluation are a core part of the claimed contribution, the paper should either provide quantitative video evaluations or explicitly limit the SOTA claim to the evaluated image tasks.
  5. [§4.4, Figure 6] The latency comparison in §4.4 states that LaVin-DiT and LVM are 'both 7B models', but LaVin-DiT is 3.4B by default (Table 3). Comparing a 3.4B diffusion model to a 7B autoregressive model confounds architecture with capacity. A matched-capacity or per-parameter latency/quality comparison is needed to support the inference-efficiency claim.
minor comments (6)
  1. [§4.2, first paragraph] 'LaVin-Dit' should be 'LaVin-DiT'.
  2. [§2, Related Work] The phrase 'from the respective of image-resembling generation' should be 'from the perspective'; there are several other grammatical issues throughout the text that should be corrected.
  3. [Algorithm 2, line 9] The final line 'Decode sample: ˆy ← Dec(z0)' is ambiguous; it should refer to the integrated latent z(0) at t=0, not the initial latent z0 defined in Algorithm 1.
  4. [Table 3] The column header 'K.V . groups' has a formatting issue; also clarify that grouped-query attention is only used for the 3.4B model (the 0.1B and 1.0B rows have no KV groups), and what value is used for the 3.4B model.
  5. [§B.2, Depth Estimation] The depth evaluation protocol mentions least-squares alignment but does not define the threshold accuracy δ1; please specify the standard thresholds (e.g., δ1 < 1.25) and whether the metrics are computed after alignment.
  6. [Figure 7] The caption of Figure 7 does not define the plotted metric or the ten tasks included; add a description so the figure is self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the model's benchmark results are not derived from its training inputs by construction, and the pseudo-label teacher provenance issue is a contamination risk, not a demonstrated circular loop.

full rationale

LaVin-DiT is an empirical systems paper whose contributions are architectural (ST-VAE, J-DiT, 3D-RoPE, in-context conditioning) and evaluated by benchmarks; it contains no first-principles derivation whose conclusion could be equivalent to its premises. The flow-matching objective in Eq. (1) regresses a velocity field v_theta(z_t,t) against the interpolation target z_0 - z_1, and neither the latent encoder/decoder nor the joint attention mechanism presupposes any of the reported benchmark outcomes. The paper cites previous work by the same authors (e.g., [34, 41, 66, 68, 70, 78]) only in related-work and in-context-learning context; no load-bearing claim rests on these self-citations, and no uniqueness theorem or earlier ansatz is imported to force the model choice. The strongest possible circularity candidate is the use of Depth-Anything V2 and Stable-Normal (turbo) to generate pseudo depth/normal labels on ImageNet-1K (Sec. 4.1, Table 4) followed by evaluation on NYUv2 (Table 2). The paper does not, however, fit any parameter to NYUv2 or define its outputs in terms of NYUv2 ground truth; the teachers are external models, and whether their training corpora included NYUv2 is a benchmark-contamination/provenance question, not a demonstrated reduction of LaVin-DiT's prediction to its training input. The internal inconsistencies noted by readers, such as the colorization MSE of 0.24 in Table 1 versus 0.273 in Figure 5 and the labeling of single-object detection as 'unseen' despite detection training data in Table 4, are correctness/reporting issues and do not establish circularity. The Limitations section further concedes that the model struggles when task definitions deviate from the training distribution, which is consistent with an empirical, non-circular claim.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

No new physical or ontological entities are proposed; ST-VAE and J-DiT are compositions of existing mechanisms. The free parameters and axioms listed above are the hand-chosen design decisions and unverified assumptions on which the benchmark claims rest.

free parameters (4)
  • Architecture hyperparameters (latent channels, hidden dim, layers, heads, KV groups, patch size) = Table 3: 16 latent channels, 2304 hidden, 22 layers, 32 heads, 4 KV groups, 2x2 patch
    Chosen by hand for the 0.1B, 1B, and 3.4B variants; the scalability claim depends on these capacities.
  • Number of inference timesteps N = 20
    Set by default in Section 4.1; no ablation over step count is reported, so the effect on output quality is unmeasured.
  • Task context length (number of input-target pairs) = 8 in latency comparison
    Section 4.4 uses 8 input-target pairs; the paper shows longer contexts improve performance, so this choice affects all reported numbers.
  • Training hyperparameters (learning rate, batch size, steps, EMA) = LR 1e-4 then 5e-5, batch 640, 100k plus 20k steps, EMA 0.9999
    Reported in Section 4.1; these choices are not derived and could affect the convergence and scaling conclusions.
assumptions (6)
  • domain assumption ST-VAE latent space preserves enough spatial-temporal detail for all dense prediction tasks.
    The whole pipeline runs in latent space (Section 3.1.1); if reconstruction is lossy, every output is ceiling-limited, but no reconstruction-quality analysis is provided.
  • domain assumption Small sets of input-target pairs fully define the task in latent space.
    Section 3.2 and Algorithm 2 rely on this for zero-fine-tuning generalization; the Limitations section admits the model struggles when task definitions deviate from training.
  • domain assumption Flow matching with 20 Euler steps produces converged samples.
    Inference uses N=20 (Section 4.1) with no step-count ablation, so convergence is assumed.
  • domain assumption Pseudo depth and normal labels from Depth-Anything V2 and Stable-Normal are accurate and do not leak evaluation-set knowledge.
    Section 4.1 generates ImageNet pseudo-labels from these teachers and Table 2 evaluates on NYUv2; overlap between teacher training data and NYUv2 is not discussed.
  • domain assumption The assembled multi-task dataset is diverse enough for in-context generalization across tasks.
    Section 4.1 and Table 4 define the training distribution; no task-interference or data-ablation analysis is given.
  • domain assumption Synthetic targets (Canny edges, crops, grayscale, Albumentations degradations) are valid supervision for the corresponding restoration tasks.
    Table 4 uses OpenCV and Albumentations to generate labels; the fidelity of these synthetic tasks to real-world degradations is not validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LaVin-DiT: Large Vision Diffusion Transformer." pith.science (2026). https://pith.science/paper/PLWW3LKE

@misc{pith2026241111505,
  author       = {Pith},
  title        = {Pith review of: LaVin-DiT: Large Vision Diffusion Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PLWW3LKE}},
  note         = {Machine review of arXiv:2411.11505}
}
read the original abstract

This paper presents the Large Vision Diffusion Transformer (LaVin-DiT), a scalable and unified foundation model designed to tackle over 20 computer vision tasks in a generative framework. Unlike existing large vision models directly adapted from natural language processing architectures, which rely on less efficient autoregressive techniques and disrupt spatial relationships essential for vision data, LaVin-DiT introduces key innovations to optimize generative performance for vision tasks. First, to address the high dimensionality of visual data, we incorporate a spatial-temporal variational autoencoder that encodes data into a continuous latent space. Second, for generative modeling, we develop a joint diffusion transformer that progressively produces vision outputs. Third, for unified multi-task training, in-context learning is implemented. Input-target pairs serve as task context, which guides the diffusion transformer to align outputs with specific tasks within the latent space. During inference, a task-specific context set and test data as queries allow LaVin-DiT to generalize across tasks without fine-tuning. Trained on extensive vision datasets, the model is scaled from 0.1B to 3.4B parameters, demonstrating substantial scalability and state-of-the-art performance across diverse vision tasks. This work introduces a novel pathway for large vision foundation models, underscoring the promising potential of diffusion transformers. The code and models are available.

Figures

Figures reproduced from arXiv: 2411.11505 by the authors.

Figure 1
Figure 1. Comparison of autoregressive and diffusion model￾ing. (a) In autoregressive modeling, visual data is divided into a sequence of patches and transformed into a one-dimensional se￾quence. The model then predicts each token sequentially from left to right and top to bottom, which is computationally intensive for high-dimensional visual data. Besides, tokens marked in red and blue illustrate disrupted spatial dependenci… view at source ↗
Figure 2
Figure 2. Overview of Large Vision Diffusion Model (LaVin-DiT). As shown in panel (a), the model initially compresses input visual data from the pixel space into a latent space, where multiple input-target pairs serve as the task context. A target is perturbed with Gaussian noise through a diffusion process. Guided by the task context and query, the Joint Diffusion Transformer (J-DiT) iteratively denoises this noisy target ov… view at source ↗
Figure 3
Figure 3. Qualitative results on diverse image and video-based tasks. The first ten rows show image-based tasks, where each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). The last four rows show video-based tasks, where each row includes a video sequence with a series of target frames as task context, followed by a query frame. A s… view at source ↗
Figures from the paper (25 more)
Figure 5
Figure 5. Figure 5: Performance for LaVin-DiT of varying sizes. Comparison of LaVin￾DiT with different parameters on coloriza￾tion (MSE) and depth estimation (AbsRel). Lower values indicate better performance [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Effect of task context length. Longer task context can consistently improve the performance of downstream tasks [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Qualitative results on object detection. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color. video surface normal estimation ( [PITH_FULL_IMAGE:figure…
Figure 9
Figure 9. Figure 9: Qualitative results on foreground segmentation. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Qualitative results on panoptic segmentation. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Qualitative results on pose estimation. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Qualitative results on pose-to-image generation. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Qualitative results on depth estimation. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Qualitative results on depth-to-image generation. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Qualitative results on surface normal estimation. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Qualitative results on normal-to-image generation. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Qualitative results on edge detection. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: Qualitative results on inpainting. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]
Figure 19
Figure 19. Figure 19: Qualitative results on image colorization. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p027_19.png]
Figure 20
Figure 20. Figure 20: Qualitative results on de-glass blur. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p028_20.png]
Figure 21
Figure 21. Figure 21: Qualitative results on de-motion blur. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p029_21.png]
Figure 22
Figure 22. Figure 22: Qualitative results on de-raining. Each row contains a sequence of images interleaved with annotations, followed by a query. The last image is predicted by the model (marked in red). Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p030_22.png]
Figure 23
Figure 23. Figure 23: Qualitative results on frame prediction. Each row includes a video sequence with a series of target frames as task context (marked in blue), followed by a query frame (marked in yellow). A set of frames in the red box indicates the model’s predictions. Due to the leng…
Figure 24
Figure 24. Figure 24: Qualitative results on video depth estimation. Each row includes a video sequence with a series of target frames as task context (marked in blue), followed by a query frame (marked in yellow). A set of frames in the red box indicates the model’s predictions. Due to th…
Figure 25
Figure 25. Figure 25: Qualitative results on depth-to-video generation. Each row includes a video sequence with a series of target frames as task context (marked in blue), followed by a query frame (marked in yellow). A set of frames in the red box indicates the model’s predictions. Due to…
Figure 26
Figure 26. Figure 26: Qualitative results on video surface normal estimation. Each row includes a video sequence with a series of target frames as task context (marked in blue), followed by a query frame (marked in yellow). A set of frames in the red box indicates the model’s predictions. …
Figure 27
Figure 27. Figure 27: Qualitative results on normal-to-video generation. Each row includes a video sequence with a series of target frames as task context (marked in blue), followed by a query frame (marked in yellow). A set of frames in the red box indicates the model’s predictions. Due t…
Figure 28
Figure 28. Figure 28: Qualitative results on optical flow estimation. Each row includes a video sequence with a series of target frames as task context (marked in blue), followed by a query frame (marked in yellow). A set of frames in the red box indicates the model’s predictions. Due to t…
Figure 29
Figure 29. Figure 29: Qualitative results on video instance segmentation. Each row includes a video sequence with a series of target frames as task context (marked in blue), followed by a query frame (marked in yellow). A set of frames in the red box indicates the model’s predictions. Due …
Figure 30
Figure 30. Figure 30: Potential application of single-view scene reconstruction. Given an RGB image and predicted depth map, we lift this image into a 3D space. We illustrate three views of this scene. Best viewed in color [PITH_FULL_IMAGE:figures/full_fig_p038_30.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

84 extracted references · 43 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In EMNLP, 2023. 4

    2023

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, pages 23716–23736, 2022. 3

    2022

  4. [4]

    Rethinking inductive biases for surface normal estimation

    Gwangbin Bae and Andrew J Davison. Rethinking inductive biases for surface normal estimation. In CVPR, pages 9535– 9545, 2024. 14

    2024

  5. [5]

    Sequential modeling enables scalable learn- ing for large vision models

    Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learn- ing for large vision models. In CVPR, pages 22861–22872,

  6. [6]

    All are worth words: A vit backbone for diffusion models

    Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR, pages 22669–22679, 2023. 3

    work page 2023

  7. [7]

    One transformer fits all distributions in multi-modal diffu- sion at scale

    Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffu- sion at scale. In ICML, pages 1692–1717, 2023. 3

    work page 2023

  8. [8]

    Visual prompting via image inpaint- ing

    Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Glober- son, and Alexei Efros. Visual prompting via image inpaint- ing. In NeurIPS, pages 25005–25017, 2022. 2, 3, 6, 7, 14

    work page 2022

Show all 84 references
  1. [9]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  2. [10]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 1, 2, 3

    2005 arXiv

  3. [11]

    End- to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End- to-end object detection with transformers. In ECCV, pages 213–229, 2020. 3

    2020

  4. [12]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 3

    arXiv

  5. [13]

    Gentron: Diffusion transformers for image and video generation

    Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Diffusion transformers for image and video generation. InCVPR, pages 6441–6451,

  6. [14]

    Pix2seq: A language modeling framework for object detection

    Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Ge- offrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109.10852, 2021. 2

    2021 arXiv

  7. [15]

    Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation

    Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In CVPR, pages 12475– 12485, 2020. 3

    2020

  8. [16]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213–3223, 2016. 5, 15

    2016

  9. [17]

    Scalable high-resolution pixel-space image syn- thesis with hourglass diffusion transformers

    Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Z Kaplan, and Enrico Shippole. Scalable high-resolution pixel-space image syn- thesis with hourglass diffusion transformers. In ICML, 2024. 3

    2024

  10. [18]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009. 5, 8, 14, 15

    2009

  11. [19]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022. 2

    2022 arXiv

  12. [20]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In ICML, 2024. 3, 4

    2024

  13. [21]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, pages 6824–6835,

  14. [22]

    Geometry- consistent generative adversarial networks for one-sided un- supervised domain mapping

    Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Bat- manghelich, Kun Zhang, and Dacheng Tao. Geometry- consistent generative adversarial networks for one-sided un- supervised domain mapping. In CVPR, pages 2427–2436,

  15. [23]

    Masked diffusion transformer is a strong image synthesizer

    Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer. In ICCV, pages 23164–23173, 2023. 3

    2023

  16. [24]

    Instructdiffusion: A generalist modeling inter- face for vision tasks

    Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, et al. Instructdiffusion: A generalist modeling inter- face for vision tasks. In CVPR, pages 12709–12720, 2024. 2

    2024

  17. [25]

    Kubric: A scalable dataset generator

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapra- gasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In CVPR, pages 3749–3761,

  18. [26]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662, 2023. 3

    2023 arXiv

  19. [27]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000–16009, 2022. 3

    2022

  20. [28]

    The kinetics hu- man action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics hu- man action video dataset. arXiv preprint arXiv:1705.06950,

    arXiv

  21. [29]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In CVPR, pages 9492–9502, 2024. 6, 8

    2024

  22. [30]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 2

    2013 arXiv

  23. [31]

    Panoptic segmentation

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Doll ´ar. Panoptic segmentation. In CVPR, pages 9404–9413, 2019. 2, 3

    2019

  24. [32]

    An im- age is worth 16x16 words: Transformers for image recogni- tion at scale

    Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weis- senborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Syl- vain Gelly, Thomas Unterthiner, and Xiaohua Zhai. An im- age is worth 16x16 words: Transformers for image recogni-...

    2021

  25. [33]

    Uvim: A unified modeling approach for vision with learned guiding codes

    Alexander Kolesnikov, Andr ´e Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes. In NeurIPS, pages 26295–26308, 2022. 2

    2022

  26. [34]

    Un- derstanding and enhancing the transferability of jailbreak- ing attacks

    Runqi Lin, Bo Han, Fengwang Li, and Tongliang Liu. Un- derstanding and enhancing the transferability of jailbreak- ing attacks. In The Thirteenth International Conference on Learning Representations, 2025. 2

    2025

  27. [35]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755, 2014. 5, 15

    2014

  28. [36]

    Feature pyramid networks for object detection

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, pages 2117–2125,

  29. [37]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023. 5, 13

    2023

  30. [38]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021. 3

    2021

  31. [39]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  32. [40]

    Unified-io: A unified model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mot- taghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In ICLR,

  33. [41]

    Deem: Diffusion models serve as the eyes of large language models for image perception

    Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting- En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, et al. Deem: Diffusion models serve as the eyes of large language models for image perception. arXiv preprint arXiv:2405.15232, 2024. 2

    2024 arXiv

  34. [42]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 3

    2024 arXiv

  35. [43]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 4195–4205, 2023. 3, 4

    2023

  36. [44]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  37. [45]

    Unicontrol: A unified diffu- sion model for controllable visual generation in the wild

    Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffu- sion model for controllable visual generation in the wild. In NeurIPS, 2024. 2

    2024

  38. [46]

    Zero: Memory optimizations toward train- ing trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward train- ing trillion parameter models. In International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16, 2020. 5

    2020

  39. [47]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125,

    arXiv

  40. [48]

    Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer

    Ren ´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623–1637, 2020. 14

    2020

  41. [49]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. In ICCV, pages 12179–12188, 2021. 6, 8

    2021

  42. [50]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2016. 2, 3

    2016

  43. [51]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, pages 10684– 10695, 2022. 2, 3, 4

    2022

  44. [52]

    One-shot learning for semantic segmentation

    Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.03410, 2017. 14

    2017 arXiv

  45. [53]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, pages 8430–8439, 2019. 5, 15

    2019

  46. [54]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, pages 746–760, 2012. 2, 6, 8, 14

    2012

  47. [55]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 ,

    arXiv

  48. [56]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  49. [57]

    Sequence to sequence learning with neural net- works

    I Sutskever. Sequence to sequence learning with neural net- works. arXiv preprint arXiv:1409.3215, 2014. 2

    2014 arXiv

  50. [58]

    Gemma: Open models based on gemini research and tech- nology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi `ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and tech- nology. arXiv preprint arXiv:2403.08295, 2024. 4

    2024 arXiv

  51. [59]

    U-dits: Downsample tokens in u-shaped diffusion transformers

    Yuchuan Tian, Zhijun Tu, Hanting Chen, Jie Hu, Chao Xu, and Yunhe Wang. U-dits: Downsample tokens in u-shaped diffusion transformers. arXiv preprint arXiv:2405.02730 ,

    arXiv

  52. [60]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1, 2

    2023 arXiv

  53. [61]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017. 2

    2017

  54. [62]

    A simple la- tent diffusion approach for panoptic segmentation and mask inpainting

    Wouter Van Gansbeke and Bert De Brabandere. A simple la- tent diffusion approach for panoptic segmentation and mask inpainting. arXiv preprint arXiv:2401.10227, 2024. 2

    2024 arXiv

  55. [63]

    Multi-task learning for dense prediction tasks: A survey

    Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3614–3633, 2021. 2

    2021

  56. [64]

    Omnitokenizer: A joint image-video tokenizer for visual generation

    Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zux- uan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation. arXiv preprint arXiv:2406.09399, 2024. 3

    2024 arXiv

  57. [65]

    Images speak in images: A generalist painter for in-context visual learning

    Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In CVPR, pages 6830–6839,

  58. [66]

    Cris: Clip-driven referring image segmentation

    Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In CVPR, pages 11686– 11695, 2022. 2

    2022

  59. [67]

    In- context learning unlocked for diffusion models

    Zhendong Wang, Yifan Jiang, Yadong Lu, Pengcheng He, Weizhu Chen, Zhangyang Wang, Mingyuan Zhou, et al. In- context learning unlocked for diffusion models. In NeurIPS, pages 8542–8562, 2023. 2

    2023

  60. [68]

    Open-vocabulary segmentation with unpaired mask-text supervision

    Zhaoqing Wang, Xiaobo Xia, Ziye Chen, Xiao He, Yandong Guo, Mingming Gong, and Tongliang Liu. Open-vocabulary segmentation with unpaired mask-text supervision. arXiv preprint arXiv:2402.08960, 2024. 2

    2024 arXiv

  61. [69]

    Finetuned language models are zero-shot learn- ers

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learn- ers. In ICLR, 2022. 2

    2022

  62. [70]

    Training-free ro- bust interactive video object segmentation

    Xiaoli Wei, Zhaoqing Wang, Yandong Guo, Chunxia Zhang, Tongliang Liu, and Mingming Gong. Training-free ro- bust interactive video object segmentation. arXiv preprint arXiv:2406.05485, 2024. 2

    2024 arXiv

  63. [71]

    Upsnet: A unified panoptic segmentation network

    Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. InCVPR, pages 8818–8826,

  64. [72]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. arXiv preprint arXiv:2406.09414, 2024. 5

    2024 arXiv

  65. [73]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 3

    2024 arXiv

  66. [74]

    Stablenormal: Reducing diffusion variance for stable and sharp normal

    Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. arXiv preprint arXiv:2406.16864, 2024. 5, 6, 8

    2024 arXiv

  67. [75]

    Colorful image colorization

    Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, pages 649–666, 2016. 2

    2016

  68. [76]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 4, 7, 14

    2018

  69. [77]

    Trained transformers learn linear models in-context

    Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. Journal of Ma- chine Learning Research, 25(49):1–55, 2024. 3

    2024

  70. [78]

    Ideal: Influence-driven selective annotations empower in-context learners in large language models

    Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling-Hao Chen, Jiale Liu, Qingyun Wu, and Tongliang Liu. Ideal: Influence-driven selective annotations empower in-context learners in large language models. In ICLR, 2024. 2

    2024

  71. [79]

    Di- ception: A generalist diffusion model for visual perceptual tasks

    Canyu Zhao, Mingyu Liu, Huanyi Zheng, Muzhi Zhu, Zhiyue Zhao, Hao Chen, Tong He, and Chunhua Shen. Di- ception: A generalist diffusion model for visual perceptual tasks. arXiv preprint arXiv:2502.17157, 2025. 2

    2025

  72. [80]

    Cv-vae: A compatible video vae for latent generative video models

    Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv-vae: A compatible video vae for latent generative video models. arXiv preprint arXiv:2405.20279, 2024. 3

    2024 arXiv

  73. [81]

    Fast training of diffusion models with masked transformers

    Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. arXiv preprint arXiv:2306.09305, 2023. 3

    2023 arXiv

  74. [82]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pages 633–641, 2017. 5, 15

    2017

  75. [83]

    Lumina-next: Making lumina-t2x stronger and faster with next-dit

    Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Lirui Zhao, Fu-Yun Wang, Zhanyu Ma, et al. Lumina-next: Making lumina-t2x stronger and faster with next-dit. arXiv preprint arXiv:2406.18583 ,

    arXiv

  76. [2024]

    More Technical Details of LaVin-DiT A.1

    2, 3 LaVin-DiT: Large Vision Diffusion Transformer Supplementary Material A. More Technical Details of LaVin-DiT A.1. Details of 3D RoPE Recall that we represent task context and query as a unified sequence of frames, which is a 3D representation. After- ward, we extend RoPE f...

    2017

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.