Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In-context video control runs 2-3x faster per diffusion step with quality intact.

desk verdict Solid engineering contribution on efficient in-context conditioning for video DiTs; the speedup is real, the selection-stability assumption is under-tested, but the paper deserves review. read the letter →

arxiv 2506.04213 v2 pith:7TR7JGFC submitted 2025-06-04 cs.CV

Xuanhua He , Quande Liu , Zixuan Ye , Weicai Ye , Qiulin Wang , Xintao Wang , Qifeng Chen , Pengfei Wan
show 2 more authors
Di Zhang Kun Gai
This is my paper
classification cs.CV
keywords in-contextconditioningvideodiffusiontransformerstokenpruningKVcachingdecoupledattentioncontrollablegenerationeditingacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In-context conditioning lets a video diffusion transformer apply many kinds of control — identity insertion, object deletion, pose, camera motion — by concatenating condition tokens with noisy video latents and processing them in one full-attention sequence, whose cost scales as $O((N_x+N_c)^2)$. This paper claims that most of that cost is wasted on two kinds of redundancy: many condition tokens carry little information, and the condition features that matter stay nearly unchanged across diffusion steps and layers. FullDiT2 removes the waste by selecting only the informative condition tokens and caching their key/value representations, and the paper reports a 2-3x per-step speedup (up to 3.43x when two dense conditions are present) with generation quality on par with or better than the full-computation baseline. If the claim holds, unified task-agnostic video conditioning becomes practical at a fraction of its previous inference cost.

What carries the argument

Two mechanisms carry the argument. Dynamic Token Selection is a lightweight learned MLP that predicts an importance score for each reference token from its value-vector features, keeps the top half of tokens for attention, and re-concatenates the dropped tokens' original hidden states after the feed-forward network so their information still reaches later blocks, cutting the attention length from $O((N_x+N_c)^2)$ toward $O((N_x+N_c/2)^2)$. Selective Context Caching has two halves: step-wise caching computes the selected reference keys and values only at the first diffusion step and reuses them at every later step, while layer-wise caching processes reference tokens only in the first layer plus the four layers with the highest Block Importance Index — a measure of how much a layer's noisy-latent output changes when the reference keys and values are included. The decoupled-attention identity — that reference self-attention plus noisy-to-all attention is exactly equivalent to full attention with the reference-to-noisy block masked — is what lets cached keys and values be reused without the training-inference mismatch that naive caching would cause.

What would settle it

Measure the cosine similarity of the selected reference tokens' key and value features between the first and later diffusion steps on a task outside the paper's set — for example, a fast camera motion or a subject whose appearance changes across the video — and compare the cached-condition model against the full-recomputation model on identity, camera-error, and pose metrics; if similarity drops markedly and task metrics degrade under caching while the baseline holds, the stability premise that the speedup depends on fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that in-context conditioning can be made two to three times cheaper per diffusion step without sacrificing control fidelity, because the conditioning signal is internally redundant in two independent ways. Token-level analysis across six tasks shows long-tail differences between consecutive reference frames and strong attention concentration — about half of the reference tokens absorb over 85% of the attention from the noisy latents — so a learned per-block selector can drop the uninformative half. The second redundancy is temporal and layer-wise: reference-token features remain stable across diffusion steps while their influence varies strongly across blocks, so the framework computes selected reference keys and values once at the first step and reuses them later, and it restricts full reference processing to the five layers with the highest Block Importance Index. To make caching safe, the paper replaces full attention with decoupled attention — reference queries attend only to reference keys while noisy queries attend to all keys — and proves this equals masked attention with the reference-to-noisy block masked off. On identity editing, re-camera, trajectory-to-video, and pose-to-video tasks, FullDiT2 reports 2.1-3.4x per-step speedups with quality metrics matching or beating the unoptimized baseline.

Load-bearing premise

The per-step speedup rests on the premise that reference-condition token features stay nearly unchanged across diffusion steps, so keys and values computed once at the first step remain valid for every later step; if those features drift under another task, sampling schedule, or longer generation, the cached conditioning would go stale and the efficiency-quality tradeoff would break.

Editorial extensions

If this is right

  • Tasks that stack dense conditions benefit most: with a reference video plus a camera trajectory, per-step latency drops to about a third of baseline and FLOPs fall to roughly 32%.
  • Because the method prunes and caches the condition pathway rather than the noisy-latent pathway, it can be composed with general DiT accelerators that reduce noisy-latent work.
  • Across the six tasks, identity similarity, pose accuracy, camera error, and text-alignment metrics stay in the baseline's range, with identity editing sometimes scoring higher than the full-computation run.
  • The decoupled-attention equivalence keeps cached inference computing the same masked-attention output the model saw in training, so the speedup does not require retraining with a different attention pattern.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The fixed configuration — top 50% token selection and five reference-processing layers — is tuned to one 28-layer pretrained model; the most direct extension is an adaptivity rule that re-selects tokens or refreshes the cache when drift is detected during sampling.
  • Because the block-importance profile is measured once on a sample set, the method presupposes that a task's layer-wise condition usage stays similar across prompts; a testable extension is measuring how the optimal layer set moves across conditioning modalities and prompt types.
  • The value-vector-based importance predictor is a proxy for attention salience; for conditions whose relevant tokens shift early versus late in the diffusion process, an attention-grounded selection signal might prove more robust.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes FullDiT2, an efficiency framework for in-context conditioning in video diffusion transformers. It combines dynamic token selection, which keeps only the top 50% of reference tokens per block, with selective context caching, which reuses reference keys and values across diffusion timesteps and skips reference processing in most transformer blocks. A decoupled attention formulation is introduced to make the cached computation match a masked-attention target. Experiments on six video editing and generation tasks report 2.1–3.4x per-step speedups with point-estimate quality metrics that are mixed but often close to baseline, plus ablations of the selection ratio, random selection, layer choice, and caching variants.

Significance. If the efficiency-quality trade-off holds, FullDiT2 is a practically useful contribution for deploying in-context conditioning video models. The paper has real strengths: the speedup numbers are direct measurements on the same hardware; the decoupled-attention equivalence proof in Appendix D is correct under standard scaled dot-product attention; the ablations in Table 2 isolate the contribution of each component; and the comparison with Delta-DiT and FORA in Appendix C.2 is informative. The main weakness is that the central claim of 'minimal degradation or even higher performance' rests on small, custom, point-estimate evaluations and on an unverified assumption that the identity of important reference tokens, not just their features, is stable across timesteps.

major comments (3)
  1. [Section 3.3.2, Figure 2d] The step-wise context caching scheme computes reference K/V only at the first diffusion timestep and reuses them for all later steps, while the Dynamic Token Selection in Section 3.3.1 is executed only at that first step. The paper supports this with the observation that reference token features are stable across timesteps, but the central trade-off also requires that the identity of the selected top-50% tokens is stable: if the important-token set shifts during denoising, the cached K/V omit tokens that later attention steps would need, and no subsequent computation recovers them. Figure 2d reports cosine similarity of reference hidden states in the pretrained full-attention model, not in the finetuned decoupled model actually used at inference, and Figure 2c shows layer-wise divergence rather than timestep-wise selection stability. Please provide a direct analysis of selection overlap across timesteps (for example, the Jaccard index of the top-k sets) or an ablation of the caching start timestep; without this, the claim that caching preserves conditioning quality is not fully supported.
  2. [Section 4.2, Tables 1 and 3; Appendix B] All quality comparisons are point estimates on small custom test sets (40 videos per ID task, 50 ReCamera pairs, and 100 pose videos, per Appendix B) and no error bars or significance tests are reported. The results are also mixed: for ID Delete, PSNR drops from 27.432 to 26.236 and SSIM from 0.903 to 0.875; for ReCamera, RotErr worsens from 1.855 to 1.924 while TransErr improves. Given the abstract claim of 'minimal degradation or even higher performance,' these fluctuations need confidence intervals or statistical tests; otherwise the quality-preservation claim is not quantitatively established.
  3. [Section 3.3.1, Eq. (3)] The importance prediction network phi takes f(Vc), but the feature transform f is never defined, and the architecture and training objective of phi are not specified. Because Dynamic Token Selection is one of the two core contributions, the manuscript should state what f is (for example, identity, value norm, or a learned projection) and how phi is trained (jointly with the finetuning or separately). This is needed for reproducibility and to assess whether the selection mechanism is actually what drives the attention concentration reported in Figure 2b.
minor comments (4)
  1. [Abstract and Section 1] There are several typos and wording issues: 'dynamical token selection' should be 'dynamic token selection'; 'identity important context tokens' should be 'identify'; 'exponential-growth computation overhead' should be 'quadratic' because attention scales quadratically with sequence length; and 'peformance' should be 'performance'.
  2. [Section 4.2 and Appendix C.2] Section 4.2 refers to 'Table 3' for the main quantitative comparison, but the main comparison table is labeled Table 1, and the appendix also contains a Table 3 for the Delta-DiT/FORA comparison. Please renumber the tables consistently.
  3. [Appendix C.3, Table 4] The theoretical attention-only speedup for the ReCamera setting is computed as about 7.57x, while the measured speedup in Table 1 is 3.43x and the GFLOPs reduction is about 3.04x. The paper should state explicitly that the appendix formulas count only self-attention token-pair interactions and explain the gap relative to measured end-to-end latency, which includes projections, MLPs, and other overheads.
  4. [Section 4.1 and Appendix A] The units 'Lat.(step/s)' and 'GFLOPs' are used inconsistently: latency is reported as seconds per step (so a lower value is better, as marked), while GFLOPs per step is a total computation value. Clarify the units in the table headers and in the text describing the efficiency measurements.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the efficiency gains are either directly measured or algebraically derived from explicit architectural choices, and no fitted quantity is repackaged as a prediction.

full rationale

FullDiT2's central efficiency claim, a 2-3x speedup per diffusion step, is not obtained by fitting a target and then reporting the fit as a prediction. The reduction chain is: FullDiT's ICC cost is quadratic in the concatenated sequence length (Section 3.2); Dynamic Token Selection halves Nc; step caching reuses reference K/V after t=0; layer caching restricts reference processing to Ls=5 layers. The Appendix C.3 cost formulas are algebraic consequences of these architectural choices with stated T, L, Ls, and the latency/FLOPs numbers in Tables 1 and 3 are direct measurements. The decoupled-attention equivalence in Appendix D is an exact algebraic identity under standard scaled dot-product attention, so it does not smuggle in the efficiency conclusion. The quality-preservation claim is empirical; no quality metric used in evaluation (CLIP-I, DINO, PSNR, PCK, etc.) is a fitted parameter of the method. The BI-based layer selection is an analysis-driven configuration choice computed on the pretrained FullDiT baseline and then ablated (Table 2); it is not a prediction derived from the reported quality metrics. FullDiT [14] is cited as the baseline model by overlapping authors, but this self-citation is not load-bearing: the baseline is an externally published architecture and all comparisons are actual measurements. The skeptical concern that cached token selection could be stale across timesteps targets the empirical validity of the stability assumption (supported only by Fig. 2d), which is a correctness/robustness risk, not a circular derivation. No step in the paper's own equations equates a claimed output with an input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on empirical stability of context features across timesteps, a learned importance predictor with unspecified features, and BI-selected layer sets tuned on training samples. The decoupled-attention proof uses standard linear algebra. No new physical entities are introduced; the efficiency gains follow from algorithmic choices plus measured FLOPs.

free parameters (4)
  • token selection ratio k/n_c = 0.5 (top 50%)
    Chosen by ablation in Table 2; 50% balances FLOPs and quality.
  • number of context-processing layers L_s = 5 (layer 0 plus top-4 BI layers)
    Selected by Block Importance Index computed on training samples; the set is fixed per task at inference.
  • importance prediction network phi (MLP weights) = learned during fine-tuning
    Learned to map features of reference Value vectors to scalar salience; f is not specified in the paper.
  • caching start timestep = t=0 (first sampling step)
    Reference K/V are computed once at the first timestep and reused for all T-1 later steps.
assumptions (4)
  • domain assumption Reference (context) token features remain approximately stable across diffusion timesteps, so K/V computed at the first step can be reused for all steps.
    Empirical observation in Figure 2(d) on one task; asserted in Section 3.3.2 and used to justify step-wise caching.
  • domain assumption Value-vector norm or simple features predict the attention relevance of a token, as used by the learned importance predictor.
    Inspired by [40]; not proven for video ICC, basis of Dynamic Token Selection in Section 3.3.1.
  • domain assumption The Block Importance Index computed on a finite set of training samples generalizes to test samples and remains valid after fine-tuning.
    Used in Section 3.3.2 and Appendix C.1 to pre-select a fixed set of 5 layers per task.
  • standard math Standard softmax attention equivalence used in the decoupled attention proof.
    Appendix D relies on softmax row-wise masking and block matrix multiplication; this is standard linear algebra.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers." pith.science (2026). https://pith.science/paper/7TR7JGFC

@misc{pith2026250604213,
  author       = {Pith},
  title        = {Pith review of: FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7TR7JGFC}},
  note         = {Machine review of arXiv:2506.04213}
}
read the original abstract

Fine-grained and efficient controllability on video diffusion transformers has raised increasing desires for the applicability. Recently, In-context Conditioning emerged as a powerful paradigm for unified conditional video generation, which enables diverse controls by concatenating varying context conditioning signals with noisy video latents into a long unified token sequence and jointly processing them via full-attention, e.g., FullDiT. Despite their effectiveness, these methods face quadratic computation overhead as task complexity increases, hindering practical deployment. In this paper, we study the efficiency bottleneck neglected in original in-context conditioning video generation framework. We begin with systematic analysis to identify two key sources of the computation inefficiencies: the inherent redundancy within context condition tokens and the computational redundancy in context-latent interactions throughout the diffusion process. Based on these insights, we propose FullDiT2, an efficient in-context conditioning framework for general controllability in both video generation and editing tasks, which innovates from two key perspectives. Firstly, to address the token redundancy, FullDiT2 leverages a dynamic token selection mechanism to adaptively identify important context tokens, reducing the sequence length for unified full-attention. Additionally, a selective context caching mechanism is devised to minimize redundant interactions between condition tokens and video latents. Extensive experiments on six diverse conditional video editing and generation tasks demonstrate that FullDiT2 achieves significant computation reduction and 2-3 times speedup in averaged time cost per diffusion step, with minimal degradation or even higher performance in video generation quality. The project page is at \href{https://fulldit2.github.io/}{https://fulldit2.github.io/}.

Figures

Figures reproduced from arXiv: 2506.04213 by the authors.

Figure 1
Figure 1. Comparison of our FullDiT2 with adapter-based methods and Full-DiT. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Empirical Analysis of Redundancy in In-Context Conditioning. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the FullDiT2 Framework. Left: The Dynamic Token Selection (DTS) module [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The qualitative comparison between FullDiT2 and baseline model on diverse tasks. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison of computational costs of self [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Training sample of our ID-related dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: The block importance on different tasks. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VideoCanvas: Unified Video Completion from Arbitrary Spatiotemporal Patches via In-Context Conditioning

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A single diffusion model with in-context conditioning and fractional RoPE positions completes videos from arbitrary spatio-temporal image patches.

Reference graph

Works this paper leans on

54 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    arXiv 2024

  2. [2]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    arXiv 2025

  3. [3]

    Open-sora 2.0: Training a commercial-level video generation model in200k.arXiv preprint arXiv:2503.09642, 2025

    Xiangyu Peng, Zangwei Zheng, Chenhui Shen, Tom Young, Xinying Guo, Binluo Wang, Hang Xu, Hongxin Liu, Mingyan Jiang, Wenjun Li, et al. Open-sora 2.0: Training a commercial-level video generation model in200k.arXiv preprint arXiv:2503.09642, 2025

    arXiv 2025

  4. [4]

    Id-animator: Zero-shot identity-preserving human video generation.arXiv preprint arXiv:2404.15275, 2024

    Xuanhua He, Quande Liu, Shengju Qian, Xin Wang, Tao Hu, Ke Cao, Keyu Yan, and Jie Zhang. Id-animator: Zero-shot identity-preserving human video generation.arXiv preprint arXiv:2404.15275, 2024

    arXiv 2024

  5. [5]

    Stylemas- ter: Stylize your video with artistic generation and translation.arXiv preprint arXiv:2412.07744, 2024

    Zixuan Ye, Huijuan Huang, Xintao Wang, Pengfei Wan, Di Zhang, and Wenhan Luo. Stylemas- ter: Stylize your video with artistic generation and translation.arXiv preprint arXiv:2412.07744, 2024

    arXiv 2024

  6. [6]

    Stylecrafter: Enhancing stylized text-to-video generation with style adapter.arXiv preprint arXiv:2312.00330, 2023

    Gongye Liu, Menghan Xia, Yong Zhang, Haoxin Chen, Jinbo Xing, Yibo Wang, Xintao Wang, Yujiu Yang, and Ying Shan. Stylecrafter: Enhancing stylized text-to-video generation with style adapter.arXiv preprint arXiv:2312.00330, 2023

    arXiv 2023

  7. [7]

    Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024

    Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024

    arXiv 2024

  8. [8]

    Sparsectrl: Adding sparse controls to text-to-video diffusion models

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. InEuropean Conference on Computer Vision, pages 330–348. Springer, 2024

    2024

Show all 54 references
  1. [9]

    Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation.arXiv preprint arXiv:2406.01900, 2024

    Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation.arXiv preprint arXiv:2406.01900, 2024

    2024 arXiv

  2. [10]

    Con- trolnext: Powerful and efficient control for image and video generation.arXiv preprint arXiv:2408.06070, 2024

    Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming-Chang Yang, and Jiaya Jia. Con- trolnext: Powerful and efficient control for image and video generation.arXiv preprint arXiv:2408.06070, 2024

    2024 arXiv

  3. [11]

    Ominicontrol: Minimal and universal control for diffusion transformer.arXiv preprint arXiv:2411.15098, 2024

    Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Minimal and universal control for diffusion transformer.arXiv preprint arXiv:2411.15098, 2024

    2024 arXiv

  4. [12]

    Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models.arXiv preprint arXiv:2502.01061, 2025

    Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang. Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models.arXiv preprint arXiv:2502.01061, 2025. 10

    2025 arXiv

  5. [13]

    Long context tuning for video generation.arXiv preprint arXiv:2503.10589, 2025

    Yuwei Guo, Ceyuan Yang, Ziyan Yang, Zhibei Ma, Zhijie Lin, Zhenheng Yang, Dahua Lin, and Lu Jiang. Long context tuning for video generation.arXiv preprint arXiv:2503.10589, 2025

    2025 arXiv

  6. [14]

    Fulldit: Multi-task video generative foundation model with full attention

    Xuan Ju, Weicai Ye, Quande Liu, Qiulin Wang, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, and Qiang Xu. Fulldit: Multi-task video generative foundation model with full attention. arXiv preprint arXiv:2503.19907, 2025

    2025 arXiv

  7. [15]

    Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024

    2024 arXiv

  8. [16]

    Follow your pose: Pose-guided text-to-video generation using pose-free videos

    Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided text-to-video generation using pose-free videos. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4117–4125, 2024

    2024

  9. [17]

    Recammaster: Camera-controlled generative rendering from a single video.arXiv preprint arXiv:2503.11647, 2025

    Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Camera-controlled generative rendering from a single video.arXiv preprint arXiv:2503.11647, 2025

    2025 arXiv

  10. [18]

    In-context lora for diffusion transformers.arXiv preprint arXiv:2410.23775, 2024

    Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, and Jingren Zhou. In-context lora for diffusion transformers.arXiv preprint arXiv:2410.23775, 2024

    2024 arXiv

  11. [19]

    Less-to- more generalization: Unlocking more controllability by in-context generation.arXiv preprint arXiv:2504.02160, 2025

    Shaojin Wu, Mengqi Huang, Wenxu Wu, Yufeng Cheng, Fei Ding, and Qian He. Less-to- more generalization: Unlocking more controllability by in-context generation.arXiv preprint arXiv:2504.02160, 2025

    2025 arXiv

  12. [20]

    Visualcloze: A universal image generation framework via visual in-context learning.arXiv preprint arXiv:2504.07960, 2025

    Zhong-Yu Li, Ruoyi Du, Juncheng Yan, Le Zhuo, Zhen Li, Peng Gao, Zhanyu Ma, and Ming- Ming Cheng. Visualcloze: A universal image generation framework via visual in-context learning.arXiv preprint arXiv:2504.07960, 2025

    2025

  13. [21]

    Insert anything: Image insertion via in-context editing in dit.arXiv preprint arXiv:2504.15009, 2025

    Wensong Song, Hong Jiang, Zongxing Yang, Ruijie Quan, and Yi Yang. Insert anything: Image insertion via in-context editing in dit.arXiv preprint arXiv:2504.15009, 2025

    2025 arXiv

  14. [22]

    Dreamo: A unified framework for image customization

    Chong Mou, Yanze Wu, Wenxu Wu, Zinan Guo, Pengze Zhang, Yufeng Cheng, Yiming Luo, Fei Ding, Shiwen Zhang, Xinghui Li, et al. Dreamo: A unified framework for image customization. arXiv preprint arXiv:2504.16915, 2025

    2025

  15. [23]

    Ominicontrol2: Efficient conditioning for diffusion transformers.arXiv preprint arXiv:2503.08280, 2025

    Zhenxiong Tan, Qiaochu Xue, Xingyi Yang, Songhua Liu, and Xinchao Wang. Ominicontrol2: Efficient conditioning for diffusion transformers.arXiv preprint arXiv:2503.08280, 2025

    2025 arXiv

  16. [24]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  17. [25]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022

    2022 arXiv

  18. [26]

    On distillation of guided diffusion models

    Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297–14306, 2023

    2023

  19. [27]

    Latent consistency models: Synthesizing high-resolution images with few-step inference.arXiv preprint arXiv:2310.04378, 2023

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference.arXiv preprint arXiv:2310.04378, 2023

    2023 arXiv

  20. [28]

    Q-diffusion: Quantizing diffusion models

    Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 17535–17545, 2023

    2023

  21. [29]

    Ld-pruner: Ef- ficient pruning of latent diffusion models using task-agnostic insights

    Thibault Castells, Hyoung-Kyu Song, Bo-Kyeong Kim, and Shinkook Choi. Ld-pruner: Ef- ficient pruning of latent diffusion models using task-agnostic insights. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 821–830, 2024

    2024

  22. [30]

    Token merging for fast stable diffusion

    Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4599–4603, 2023. 11

    2023

  23. [31]

    Importance-based token merging for diffusion models.arXiv preprint arXiv:2411.16720, 2024

    Haoyu Wu, Jingyi Xu, Hieu Le, and Dimitris Samaras. Importance-based token merging for diffusion models.arXiv preprint arXiv:2411.16720, 2024

    2024 arXiv

  24. [32]

    Fora: Fast- forward caching in diffusion transformer acceleration.arXiv preprint arXiv:2407.01425, 2024

    Pratheba Selvaraju, Tianyu Ding, Tianyi Chen, Ilya Zharkov, and Luming Liang. Fora: Fast- forward caching in diffusion transformer acceleration.arXiv preprint arXiv:2407.01425, 2024

    2024 arXiv

  25. [33]

    delta-dit: A training-free acceleration method tailored for diffusion transformers.arXiv preprint arXiv:2406.01125, 2024

    Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, and Tao Chen. delta-dit: A training-free acceleration method tailored for diffusion transformers.arXiv preprint arXiv:2406.01125, 2024

    2024 arXiv

  26. [34]

    Learning-to-cache: Accelerating diffusion transformer via layer caching.Advances in Neural Information Processing Systems, 37:133282–133304, 2024

    Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Learning-to-cache: Accelerating diffusion transformer via layer caching.Advances in Neural Information Processing Systems, 37:133282–133304, 2024

    2024

  27. [35]

    Relactrl: Relevance-guided efficient control for diffusion transformers.arXiv preprint arXiv:2502.14377, 2025

    Ke Cao, Jing Wang, Ao Ma, Jiasong Feng, Zhanjie Zhang, Xuanhua He, Shanyuan Liu, Bo Cheng, Dawei Leng, Yuhui Yin, et al. Relactrl: Relevance-guided efficient control for diffusion transformers.arXiv preprint arXiv:2502.14377, 2025

    2025

  28. [36]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  29. [37]

    Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  30. [38]

    Llm pruning and distillation in practice: The minitron approach.arXiv preprint arXiv:2408.11796, 2024

    Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng, Zijia Chen, Yoshi Suhara, Shizhe Diao, et al. Llm pruning and distillation in practice: The minitron approach.arXiv preprint arXiv:2408.1...

    2024 arXiv

  31. [39]

    Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    2022

  32. [40]

    Attention score is not all you need for token importance indicator in KV cache reduction: Value also matters

    Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in KV cache reduction: Value also matters. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Method...

    2024

  33. [41]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  34. [42]

    Using k-poselets for detecting people and localizing their keypoints

    Georgia Gkioxari, Bharath Hariharan, Ross Girshick, and Jitendra Malik. Using k-poselets for detecting people and localizing their keypoints. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3582–3589, 2014

    2014

  35. [43]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  36. [44]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. InProceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023

    2023

  37. [45]

    inserted

    Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, et al. Koala-36m: A large-scale video dataset improving con- sistency between fine-grained conditions and video content.arXiv preprint arXiv:2410.08260, 2024....

    arXiv

  38. [46]

    Attention is computed only over theN x noisy latent tokens for allLlayers andTtimesteps

    Baseline DiT (No Condition):In this standard configuration, no condition tokens are processed. Attention is computed only over theN x noisy latent tokens for allLlayers andTtimesteps. Cno_cond =T LN2 x (10)

  39. [47]

    All L layers and T timesteps perform attention over the total sequence length(N x +N c)≈3N x

    Baseline ICC (FullDiT-style):Here, the full set of Nc reference condition tokens are concatenated with Nx noisy latent tokens. All L layers and T timesteps perform attention over the total sequence length(N x +N c)≈3N x. Cbaseline_icc ≈T L(3Nx)2 = 9T LN2 x (11)

  40. [48]

    16 At the first timestep (t= 1 ), reference K/V must be computed

    FullDiT2 (Ours):Our method incorporates Dynamic Token Selection (reducing effective reference length to N ′ c ≈N x), step-wise context caching for reference tokens (computed at t= 1 , reused for T−1steps), and layer-wise context caching (L s layers process reference,L−L s laye...

  41. [49]

    Caching: Step:No Dynamic Token Selection is applied ( Nc ≈2N x)

    Baseline + Sel. Caching: Step:No Dynamic Token Selection is applied ( Nc ≈2N x). All L layers process reference tokens. Reference K/V are computed at t= 1 and reused for the subsequent T−1steps. Cstep_cache ≈L(7N 2 x ) + (T−1)L(3N 2 x ) = (3T+ 4)LN 2 x (15)

  42. [50]

    Caching: Layer:No Dynamic Token Selection ( Nc ≈2N x) and no step-wise context caching

    Baseline+Sel. Caching: Layer:No Dynamic Token Selection ( Nc ≈2N x) and no step-wise context caching. Reference tokens are processed in Ls layers, and skipped in L−L s layers, for all T timesteps. Clayer_cache ≈T Ls(3Nx)2 + (L−L s)N 2 x =T(L+ 8L s)N 2 x (16)

  43. [51]

    All L layers and T timesteps process the reduced sequence length (Nx +N ′ c)≈2N x

    Baseline + Dynamic Token Selection:Only Dynamic Token Selection is active (effective reference length N ′ c ≈N x). All L layers and T timesteps process the reduced sequence length (Nx +N ′ c)≈2N x. No stepwise caching or layer caching. Cdts_only ≈T L(Nx +N x)2 = 4T LN2 x (17)

  44. [52]

    Context Caching are active

    Ours w/o Dynamic Token Selection:FullDiT2 framework but with Dynamic Token Selection disabled (Nc ≈2N x is used instead ofN ′ c ≈N x). Context Caching are active. Cours_no_dts ≈(L+ 6L s)N 2 x + (T−1)(L+ 2L s)N 2 x = (T L+ (2T+ 4)Ls)N 2 x (18)

  45. [53]

    Reference K/V are recomputed at every step

    Ours w/o Selective Caching: Step:FullDiT2 framework but with step-wise context caching disabled. Reference K/V are recomputed at every step. The cost of the first step is effectively repeated Ttimes. Cours_no_temporal ≈T Ls(3N 2 x ) + (L−L s)N 2 x =T(L+ 2L s)N 2 x (19)

  46. [54]

    Ours w/o Selective Caching: Layer ( Ls →L ):FullDiT2 framework but with layer-wise context caching disabled; allLlayers process reference tokens. Cours_no_layer ≈3LN 2 x + (T−1)2LN 2 x = (2T+ 1)LN 2 x (20) Summary Table of Computational Costs and Speedup RatiosFinally, we summ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.