Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

Chao Xu; Cuifeng Shen; Hanlin Tang; Kan Liu; Lin Qu; Maohua Li; Qirui Li; Shao-Qun Zhang; Tao Lan; Yanke Zhou

REVIEW 2 major objections 4 minor 60 references

The paper claims chat-template tokens, although content-free at the encoder, become implicit semantic registers inside text-to-image diffusion transformers: they absorb most image-to-text attention and causally maintain object identity thro

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:19 UTC pith:AMVEVQHE

load-bearing objection Good causal toolkit, solid descriptive findings; the two-stage circuit timing rests on a masking confound and the paper's own appendix qualifies the register claim. the 2 major comments →

arxiv 2607.19139 v2 pith:AMVEVQHE submitted 2026-07-21 cs.CV

Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

Maohua Li , Qirui Li , Yanke Zhou , Yiduo Li , Zhaosheng Chi , Chao Xu , Cuifeng Shen , Yixuan Xu

show 5 more authors

Hanlin Tang Kan Liu Tao Lan Lin Qu Shao-Qun Zhang

This is my paper

classification cs.CV

keywords diffusion transformersattention sinkssemantic registerstext-to-image generationjoint attentionchat templatescausal interventionshead pruning

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to establish that text conditioning in modern text-to-image diffusion transformers is not a direct, layer-by-layer reading of the prompt tokens. Instead, the formatting tokens added by the chat template—while nearly content-free at the text encoder—become 'implicit semantic registers': image tokens route most of their text attention to these tokens, and causal interventions show the template tokens maintain object identity during generation. The claimed mechanism is indirect: in the first block, prompt semantics are written into the image latents; in the second block, the template tokens read that identity back from the image stream; deeper layers then attend to the template tokens rather than to the original prompt. If correct, this overturns the direct-injection picture of conditioning and yields a training-free pruning rule that removes causally inert prompt-reading heads, saving about 20% of joint-attention FLOPs with a small accuracy cost on an object-focused benchmark.

Core claim

At the text encoder's output, chat-template tokens carry little prompt-specific information; replacing them with a prompt-agnostic average leaves most generated objects unchanged. Yet inside the DiT's joint attention, image queries direct most of their text-stream attention to these template tokens, with a single delimiter often the most-attended text token. Cross-trajectory head transplants show that heads reading the prompt span are causally inert, while transplanting heads that barely read the prompt flips object identity after roughly 18% of heads. Masking template tokens' attention to prompt tokens leaves identity intact; masking their attention to image latents erases it within the fir

What carries the argument

The central object is the joint attention matrix of a multimodal diffusion transformer, partitioned into text-to-text, text-to-image, image-to-text (I2T), and image-to-image blocks. The paper's key construct is the 'implicit semantic register': a content-free chat-template token that stores object identity inside the model. The load-bearing identity is the indirect circuit S→I→R—prompt semantics enter image latents in the first block, template tokens read that identity back in the second block, and deeper image layers attend to the template tokens. The causal framework combines token-span decomposition (semantic span S vs template span R), cross-prompt swaps and template averaging, cross-tra

Load-bearing premise

The causal story hinges on pre-softmax attention masking in the layer-wise experiment: because masking one key span renormalizes the remaining keys, the observed identity collapse when the image-to-template route is masked could stem from attention redistribution to other spans rather than from deleting the register's image-reading path.

What would settle it

Run the layer-wise masking experiment with a mass-preserving control—halving or zeroing the targeted span's scores while rescaling the remaining scores so each row's total attention stays fixed. If the object no longer collapses when the image-reading route is blocked under that control, the claimed S→I→R circuit would be an artifact of softmax renormalization; if identity still collapses, the circuit is confirmed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Text conditioning in joint-attention DiTs must be understood as an indirect register loop, so future interpretability and editing methods should target the S→I→R pathway rather than assuming prompt tokens are read at every layer.
Object identity is committed in early blocks, carried by middle blocks, and refined in late blocks, giving a concrete depth-based map for surgical interventions.
Heads that attend strongly to the prompt span are causally inert for identity transfer; pruning them training-free over the late denoising steps removes about 20% of joint-attention FLOPs with only a 1.4-point drop on an object-focused benchmark in the studied model.
The register role persists across prompt complexity, language, few-step distillation, and models with extra conditioning modalities, suggesting the mechanism is intrinsic to the joint-attention computation rather than a sampling or template artifact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If template tokens are the main identity carriers in late layers, sparse-attention or KV-cache schemes could keep only the register keys and drop most prompt-content keys after early blocks, potentially exceeding the reported 20% savings.
The S→I→R ordering implies the first block is a bottleneck: edits or interventions applied after the second block may fail to change identity unless they also rewrite the image latents that registers read back, suggesting a testable design rule for editing methods.
Because the register is learned rather than encoded, adding explicit register tokens with a training objective might make chat-template registers unnecessary, enabling more controllable conditioning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Good causal toolkit, solid descriptive findings; the two-stage circuit timing rests on a masking confound and the paper's own appendix qualifies the register claim.

read the letter

Two things to know. This is the first paper I have seen that causally isolates chat-template tokens as semantic registers in an LLM-conditioned DiT, and it puts a genuinely useful toolkit on the table: token decomposition, span interventions, head transplants, layer-wise masking. Second, the precise two-stage timing story (S→I in block 1, I→R in block 2) is not as solid as the abstract makes it sound. The masking evidence has a renormalization confound, and the authors' own Appendix E.8 backs off from the strong register-only claim.

The paper does a lot right. The descriptive discovery — that image tokens pour attention into content-free delimiter tokens rather than the prompt — is clean and replicated over three benchmarks, two Qwen checkpoints, FLUX.2, Krea-2-Turbo, a distilled checkpoint, and an editing model. The head-swap experiment is a nice piece of evidence: transplanting heads ranked by their reading of the semantic span is causally inert until you swap in heads that barely read S. That decoupling is real and is the paper's best contribution. The pruning rule, while first-cut, is prompt-independent, training-free, and evaluated on GenEval. They ship code. No fitted parameters make the central claim true; the core phenomenon is a descriptive fact about attention.

The soft spots are in proportion. The masking experiment in Section 4.3 is the only place where they claim the specific circuit timing. Hard-masking R→I before softmax does not delete the route cleanly; it redistributes the attention mass that would have gone to the image span over the remaining text keys, which changes the entire attention profile of the register tokens. The R→S control is not magnitude-matched: the semantic span is a couple of tokens, while the image span is large, so the two conditions are not symmetric perturbations. A value-ablation control — replacing the image keys' values rather than zeroing attention — would settle whether the collapse is due to losing an information route or to generic renormalization. The paper does not do that. Also, Table 2 has no error bars; the 1.4-point GenEval drop is a single-run number.

The paper's own appendix qualifies the claim in a way the abstract does not. Appendix E.8 shows a text-side per-span swap never reproduces the target object; swapping R alone pushes toward the unconditional default, and the authors conclude that part of the object's semantics is carried by the semantic span S and, in particular, the image tokens, rather than by the template register R alone. So the headline 'implicit semantic registers' is overstated if read as 'the registers alone carry identity.'

This is a paper for people working on mechanistic interpretability of diffusion transformers and on DiT efficiency. It deserves a serious referee. I would ask the authors to add a value-ablation control for the masking, report error bars for pruning, and revise the abstract to match the Appendix E.8 qualification. None of that is fatal; the descriptive findings and the head-swap result stand.

Referee Report

2 major / 4 minor

Summary. The paper studies how text conditioning is incorporated inside a chat-templated MMDiT (Qwen-Image). By separating conditioning tokens into a semantic span S and a chat-template span R, it shows that R tokens receive a dominant share of image-to-text attention, carry little prompt-specific content at the encoder output, yet act as causally load-bearing 'implicit semantic registers.' The proposed mechanism is an indirect circuit S→I→R: prompt semantics are injected into image latents in block 1, template tokens read identity back from image latents in block 2, and deeper image layers attend to R. The paper also reports a division of labor across heads and depth, and a training-free pruning rule that removes high-mS heads, cutting 20% of joint-attention FLOPs with a 1.4-point GenEval drop. Evidence includes attention statistics across benchmarks, cross-prompt swaps, template averaging, progressive head transplants, attention masking, per-span swaps, and cross-model replications on FLUX.2 and Krea-2-Turbo.

Significance. The descriptive finding that chat-template tokens become dominant attention sinks and are causally involved in identity maintenance is novel and, if correct, would revise the direct-injection view of text conditioning in DiTs. The paper's strengths include the scale of the attention statistics (multiple checkpoints and benchmarks), the complementary intervention suite, the cross-model replication in the appendix, the training-free pruning rule evaluated on GenEval, and the public code release. The attention-sink phenomenon and the causal relevance of R tokens are convincingly supported. However, the paper's signature mechanistic claim — an S→I→R circuit with block-1/block-2 timing — rests on a masking experiment that is subject to a softmax-renormalization confound and on an inference that the reported null result does not support.

major comments (2)

[§4.3, Fig. 5b] The I→R leg is not causally isolated. Hard-masking R→I before softmax removes a large key span (nimg ≈ 6,896 image keys for this model) and forces attention to renormalize over the small remaining text span (|S|=2 for 'An apple'), whereas the R→S control removes only two keys. The collapse of identity in the R→I condition could therefore be driven by the magnitude of attention redistribution rather than by deleting the image-reading route. This is load-bearing for the abstract's central claim of an indirect circuit. Please add a value-ablation control (e.g., replace the image values attended by R with a constant/zero vector while preserving attention), or a matched-mass control (mask an image span of comparable size that does not carry the target identity), and report a quantitative identity metric over a prompt set rather than only selected images.
[§4.3, Fig. 5b] The timing claim 'S→I completes within the first block, I→R completes in the second' is not established by the reported null result. Masking R→I in block 1 alone has little effect, but this is expected regardless of when S→I occurs, because R can read the image in any later block; the mask targets R→I, not I→S. To support the block-1 S→I claim, the authors would need to ablate image queries' attention to S in block 1 (I→S) or directly show that image latents after block 1 carry identity. As it stands, the experiment only shows that R does not need image access in block 1 if image access is available in a later block.

minor comments (4)

[§4.2 / Abstract] The abstract says template tokens carry 'little prompt-specific information,' while §4.2 concludes they carry semantics 'only weakly.' The averaging experiment shows that replacing R with a prompt-agnostic average can drastically change the object at CFG=1 in a nontrivial tail of prompts, and one cross-prompt swap on Qwen-Image-2512 gives DINO sim ≈ 0.54. Please align the abstract's wording with the caveated conclusion.
[Table 1] The header 'mR/mR+mS' is ambiguous. It should be written as m_R/(m_R+m_S), or defined explicitly as the fraction of I2T attention landing on the template span.
[§4.3 / Appendix E.8] The masking and head-swap experiments in §4.3 are run on a 2-step distilled LoRA at a single denoising step, while the attention statistics in Table 1 are on the base model. Appendix E.3 reports sink statistics for the distilled model, but not the masking/timing experiments. Please state whether the block-1/block-2 timing replicates on the base model, or at least discuss the generality.
[Fig. 5 and Fig. 8] The identity-transfer and masking conclusions are presented through selected qualitative examples. A quantitative identity metric (e.g., DINO similarity to the unperturbed baseline A, averaged over a prompt set) would make the causal claims easier to assess and would strengthen the paper.

Circularity Check

0 steps flagged

No significant circularity: causal claims rest on independent interventions; pruning is validated on external benchmarks; self-citations are background only.

full rationale

The paper's central claim—that chat-template tokens act as implicit semantic registers via an indirect S→I→R circuit—is supported by direct causal interventions (attention-mass measurements, cross-prompt template swaps, template averaging, cross-trajectory head transplantation, per-span progressive swaps, and layer-wise attention masking) rather than by fitting a parameter to the conclusion. The pruning rule is derived from an attention-based ranking (¯mS) and is then evaluated on external benchmarks (GenEval, LPIPS, HPSv3); the ranking is not fit to those benchmark scores, so the reported 1.4-point GenEval drop is an independent empirical outcome, not a prediction forced by construction. The only self-citations ([48], [56], [60]) appear in related-work discussions of sparse attention and DiT architecture, and none is load-bearing for the paper's causal or pruning claims. The reader-identified concern about §4.3's masking experiment—that pre-softmax masking renormalizes attention over remaining keys, so the R→I collapse may partly reflect attention redistribution—is a legitimate correctness/confound risk, but it is not circularity: the masking intervention is not definitionally equivalent to the S→I→R conclusion, and the paper's other evidence (head swaps, per-span swaps, depth localization) provides independent support. No self-definitional reduction, fitted-input-as-prediction, imported uniqueness theorem, or ansatz-smuggled-via-citation pattern is present.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central claim requires no fitted constants: it is an empirical claim about a fixed pretrained model. However, several manual thresholds enter the experimental protocol, and the causal conclusions depend on interpretive assumptions about attention interventions. 'Implicit semantic register' is a post-hoc conceptual label, not an independently testable entity.

free parameters (4)

Late-step pruning fraction f = 0.2
Manual choice in §4.5; the FLOP-reduction claim scales with f and the pruning budget.
Pruning budget K = 360 of 1,440 heads
Hand-selected operating point for the headline 20%-FLOP claim; the paper notes K=216 gives a better perceptual trade-off, so K is not derived from first principles.
Cumulative masking window k = first 2 blocks
Supports the S→I-in-block-1 and I→R-in-block-2 timing conclusion in §4.3; the exact timing claims depend on this manual truncation.
CFG guidance scale = 1 and 4
Template-averaging results in Fig. 4b are strongly CFG-dependent; the conclusion that template tokens carry little semantics only holds under the chosen guidance values.

axioms (4)

domain assumption Causal interpretation of attention interventions: transplanting (q,k,v) projections and masking attention edges captures the causal role of the targeted heads or edges.
The entire register-head and division-of-labor conclusion in §4.3-4.4 rests on this; if interventions cause off-target effects, the mechanism does not follow.
domain assumption DINOv3 CLS cosine similarity is a valid proxy for semantic identity preservation.
Used as the quantitative similarity measure in cross-prompt swaps and template-averaging experiments; no human validation is reported.
domain assumption GenEval object-correctness accuracy is a reliable measure of identity maintenance for the pruning evaluation.
The pruning rule is evaluated on a single benchmark and a single checkpoint with no error bars or repeated seeds.
domain assumption The split of conditioning into semantic span S and template span R matches the encoder functional roles.
Defined in §3.2; later experiments show template tokens carry weak semantics, so the split is approximately valid but not exact.

invented entities (1)

Implicit semantic register no independent evidence
purpose: Conceptual role assigned to chat-template tokens that are claimed to causally maintain object identity during denoising.
Not a separately posed entity; it is an interpretation of existing tokens inferred from the same causal interventions that support the paper's claims. There is no independent falsifiable handle beyond this paper's experiments.

pith-pipeline@v1.3.0-alltime-deepseek · 19735 in / 14291 out tokens · 142169 ms · 2026-08-01T13:19:55.329837+00:00 · methodology

0 comments

read the original abstract

Modern text-to-image diffusion transformers (DiTs) generate images through joint attention, in which text and image tokens interact directly within a single sequence. In large-scale DiTs, the conditioning input contains not only the user prompt but also chat-template tokens introduced by LLM-based text encoders. Yet how these tokens participate in the denoising computation remains poorly understood. To probe this, we introduce a causal interpretability framework. Using it to separate prompt-content tokens from chat-template tokens, we find that the template tokens carry little prompt-specific information at the encoder output. Yet surprisingly, they emerge as dominant image-to-text attention sinks and causally maintain object identity inside the DiT, acting as implicit semantic registers. We show that they acquire this identity indirectly. Rather than reading the prompt tokens, they draw the identity from the image latents into which the prompt semantics have already been injected at the very first layer. We further reveal a division of labor across heads and depth in DiTs, where distinct heads route semantics or render visual structure, and identity is committed in early blocks, carried by middle blocks, and refined in late ones. As a practical payoff, this analysis yields a training-free pruning rule that removes the causally inert prompt-reading heads and cuts $20\%$ of joint-attention FLOPs at a $1.4$-point cost in GenEval accuracy. Overall, our work not only reveals that the tokens encoding semantics at the input need not be those that maintain them during generation, but also provides a causal view of internal mechanisms in diffusion transformers.

discussion (0)

Reference graph

Works this paper leans on

60 extracted references · 23 linked inside Pith

[1]

Krea 2.https://www.krea.ai/blog/krea-2-technical-report, 2026

Krea AI. Krea 2.https://www.krea.ai/blog/krea-2-technical-report, 2026

2026
[2]

Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

Pith/arXiv arXiv 2025
[3]

Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.ar...

Pith/arXiv arXiv 2025
[4]

All are worth words: A vit backbone for diffusion models

Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22669–22679, 2023

2023
[5]

Why do llms attend to the first token?arXiv preprint arXiv:2504.02732, 2025

Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do llms attend to the first token?arXiv preprint arXiv:2504.02732, 2025

Pith/arXiv arXiv 2025
[6]

Pixart-σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. InEuropean Conference on Computer Vision, pages 74–91. Springer, 2024

2024
[7]

Revealing the attention floating mechanism in masked diffusion models.arXiv preprint arXiv:2601.07894, 2026

Xin Dai, Pengcheng Huang, Zhenghao Liu, Shuo Wang, Yukun Yan, Chaojun Xiao, Yu Gu, Ge Yu, and Maosong Sun. Revealing the attention floating mechanism in masked diffusion models.arXiv preprint arXiv:2601.07894, 2026

arXiv 2026
[8]

Vision transformers need registers

Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In International conference on learning representations, volume 2024, pages 2632–2652, 2024

2024
[9]

Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

2021
[10]

Scaling rectified flow transformers for high-resolution image synthesis

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning, 2024

2024
[11]

Mdtv2: Masked diffusion transformer is a strong image synthesizer.arXiv preprint arXiv:2303.14389, 2023

Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is a strong image synthesizer.arXiv preprint arXiv:2303.14389, 2023

Pith/arXiv arXiv 2023
[12]

Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

2023
[13]

When attention sink emerges in language models: An empirical view

Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. InInternational Conference on Learning Representations, volume 2025, pages 97114–97144, 2025

2025
[14]

Conceptattention: Diffusion transformers learn highly interpretable features.arXiv preprint arXiv:2502.04320, 2025

Alec Helbling, Tuna Han Salih Meral, Ben Hoover, Pinar Yanardag, and Duen Horng Chau. Conceptattention: Diffusion transformers learn highly interpretable features.arXiv preprint arXiv:2502.04320, 2025

Pith/arXiv arXiv 2025
[15]

Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

Pith/arXiv arXiv 2022
[16]

Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

2020
[17]

Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024. 10

Pith/arXiv arXiv 2024
[18]

Diffusion transformers use sink registers

Amna Jamal, Mika Tan, Clarissa Aurelia Nahid Saputra, Quan Huynh, Kevin Zhu, and Antonio Mari. Diffusion transformers use sink registers. InSecond Workshop on XAI4Science: From Understanding Model Behavior to Discovering New Scientific Knowledge, 2025. URLhttps://openreview.net/forum?id=Fc7s3UrQr1

2025
[19]

Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.Advances in Neural Information Processing Systems, 37:52481–52515, 2024

Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.Advances in Neural Information Processing Systems, 37:52481–52515, 2024

2024
[20]

Vision transformers don’t need trained registers

Nicholas Jiang, Amil Dravid, Alexei Efros, and Yossi Gandelsman. Vision transformers don’t need trained registers. Advances in neural information processing systems, 38:56557–56595, 2026

2026
[21]

See what you are told: Visual attention sink in large multimodal models.arXiv preprint arXiv:2503.03321, 2025

Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models.arXiv preprint arXiv:2503.03321, 2025

Pith/arXiv arXiv 2025
[22]

Flux.https://github.com/black-forest-labs/flux, 2024

Black Forest Labs. Flux.https://github.com/black-forest-labs/flux, 2024

2024
[23]

FLUX.2.https://bfl.ai/blog/flux-2, 2025

Black Forest Labs. FLUX.2.https://bfl.ai/blog/flux-2, 2025

2025
[24]

Remarkable robustness of llms: Stages of inference? Advances in Neural Information Processing Systems, 38:130050–130083, 2026

Vedang Lad, Jin Hwa Lee, Wes Gurnee, and Max Tegmark. Remarkable robustness of llms: Stages of inference? Advances in Neural Information Processing Systems, 38:130050–130083, 2026

2026
[25]

Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference.arXiv preprint arXiv:2502.20766, 2025

Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference.arXiv preprint arXiv:2502.20766, 2025

Pith/arXiv arXiv 2025
[26]

Qwen-image-bench: From generation to creation in text-to-image evaluation.arXiv preprint arXiv:2605.28091, 2026

Niantong Li, Guangzheng Hu, Weixu Qiao, Ying Ba, Qichen Hong, Shijun Shen, Jinlin Wang, Fan Zhou, Jianye Kang, Xin Shang, et al. Qwen-image-bench: From generation to creation in text-to-image evaluation.arXiv preprint arXiv:2605.28091, 2026

Pith/arXiv arXiv 2026
[27]

Snapkv: Llm knows what you are looking for before generation.Advances in Neural Information Processing Systems, 37:22947–22970, 2024

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation.Advances in Neural Information Processing Systems, 37:22947–22970, 2024

2024
[28]

Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

Pith/arXiv arXiv 2022
[29]

Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025

Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025

Pith/arXiv arXiv 2025
[30]

Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time.Advances in Neural Information Processing Systems, 36:52342–52364, 2023

Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time.Advances in Neural Information Processing Systems, 36:52342–52364, 2023

2023
[31]

Rethinking cross-modal interaction in multimodal diffusion transformers

Zhengyao Lv, Tianlin Pan, Chenyang Si, Zhaoxi Chen, Wangmeng Zuo, Ziwei Liu, and Kwan-Yee K Wong. Rethinking cross-modal interaction in multimodal diffusion transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 5934–5943, 2025

2025
[32]

Hpsv3: Towards wide-spectrum human preference score

Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15086–15095, 2025

2025
[33]

DiffSynth-Studio

ModelScope. DiffSynth-Studio. GitHub repository, 2024.https://github.com/modelscope/DiffSynth-Studio

2024
[34]

Improved denoising diffusion probabilistic models

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. InInternational conference on machine learning, pages 8162–8171. PMLR, 2021

2021
[35]

Scalable diffusion models with transformers

William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

2023
[36]

Sdxl: Improving latent diffusion models for high-resolution image synthesis

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. InInternational Conference on Learning Representations, volume 2024, pages 1862–1874, 2024

2024
[37]

Attention sinks and compression valleys in llms are two sides of the same coin.arXiv preprint arXiv:2510.06477, 2025

Enrique Queipo-de Llano, Álvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael Bronstein, Yann LeCun, and Ravid Shwartz-Ziv. Attention sinks and compression valleys in llms are two sides of the same coin.arXiv preprint arXiv:2510.06477, 2025. 11

arXiv 2025
[38]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PMLR, 2021

2021
[39]

Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

2020
[40]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

2022
[41]

Attention sinks in diffusion language models.arXiv preprint arXiv:2510.15731, 2025

Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri, Simone Scardapane, and Alessio Devoto. Attention sinks in diffusion language models.arXiv preprint arXiv:2510.15731, 2025

arXiv 2025
[42]

Exploring multimodal diffusion transformers for enhanced prompt-based image editing

Joonghyuk Shin, Alchan Hwang, Yujin Kim, Daneul Kim, and Jaesik Park. Exploring multimodal diffusion transformers for enhanced prompt-based image editing. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19492–19502, 2025

2025
[43]

Dinov3.arXiv preprint arXiv:2508.10104, 2025

Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025

Pith/arXiv arXiv 2025
[44]

Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

Pith/arXiv arXiv 2010
[45]

Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

Pith/arXiv arXiv 2011
[46]

Registers matter for pixel-space diffusion transformers.arXiv preprint arXiv:2605.16147, 2026

Nikita Starodubcev, Ilia Sudakov, Ilya Drobyshevskiy, Artem Babenko, and Dmitry Baranchuk. Registers matter for pixel-space diffusion transformers.arXiv preprint arXiv:2605.16147, 2026

Pith/arXiv arXiv 2026
[47]

Massive activations in large language models.arXiv preprint arXiv:2402.17762, 2024

Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models.arXiv preprint arXiv:2402.17762, 2024

Pith/arXiv arXiv 2024
[48]

Razorattention: Efficient kv cache compression through retrieval heads

Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Danning Ke, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Razorattention: Efficient kv cache compression through retrieval heads. InInternational Conference on Learning Representations, volume 2025, pages 16632–16646, 2025

2025
[49]

Quest: Query-aware sparsity for efficient long-context llm inference.arXiv preprint arXiv:2406.10774, 2024

Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference.arXiv preprint arXiv:2406.10774, 2024

Pith/arXiv arXiv 2024
[50]

Qwen-Image-2512-Turbo-LoRA-2-Steps

Wuli Team. Qwen-Image-2512-Turbo-LoRA-2-Steps. Hugging Face model repository, 2026.https://huggingface. co/Wuli-art/Qwen-Image-2512-Turbo-LoRA-2-Steps

2026
[51]

Fasa: Frequency-aware sparse attention.arXiv preprint arXiv:2602.03152, 2026

Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, and Julian McAuley. Fasa: Frequency-aware sparse attention.arXiv preprint arXiv:2602.03152, 2026

arXiv 2026
[52]

Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025

Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025

Pith/arXiv arXiv 2025
[53]

Attention sinks in diffusion transformers: A causal analysis.arXiv preprint arXiv:2605.09313, 2026

Fangzheng Wu and Brian Summa. Attention sinks in diffusion transformers: A causal analysis.arXiv preprint arXiv:2605.09313, 2026

Pith/arXiv arXiv 2026
[54]

Efficient streaming language models with attention sinks

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InInternational Conference on Learning Representations, volume 2024, pages 21875–21895, 2024

2024
[55]

Duoattention: Efficient long-context llm inference with retrieval and streaming heads

Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. InInternational Conference on Learning Representations, volume 2025, pages 37228–37253, 2025

2025
[56]

Rethinking cross-layer information routing in diffusion transformers.arXiv preprint arXiv:2605.20708, 2026

Chao Xu, Maohua Li, Qirui Li, Yixuan Xu, Yanke Zhou, Yunhe Li, Cuifeng Shen, Hanlin Tang, Kan Liu, Tao Lan, et al. Rethinking cross-layer information routing in diffusion transformers.arXiv preprint arXiv:2605.20708, 2026

Pith/arXiv arXiv 2026
[57]

On the nature of attention sink that shapes decoding strategy in mllms.arXiv preprint arXiv:2603.14337, 2026

Suho Yoo, Youngjoon Jang, and Joon Son Chung. On the nature of attention sink that shapes decoding strategy in mllms.arXiv preprint arXiv:2603.14337, 2026. 12

Pith/arXiv arXiv 2026
[58]

The unreasonable effectiveness of deep features as a perceptual metric

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018

2018
[59]

H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

2023
[60]

that”,[/SYSTEM_PROMPT], and <s> as the leading sinks and “ that

Yanke Zhou, Yiduo Li, Hanlin Tang, Maohua Li, Kan Liu, Lan Tao, Lin Qu, Yuan Yao, and Xiaoxing Ma. Full attention strikes back: Transferring full attention into sparse within hundred training steps.arXiv preprint arXiv:2605.16928, 2026. 13 Appendix Causal Interpretability Framework for DiTs MM-ATTN Block Head Head Head Head Block Sink Heads 01 02 03 04 I2...

Pith/arXiv arXiv 2026

[1] [1]

Krea 2.https://www.krea.ai/blog/krea-2-technical-report, 2026

Krea AI. Krea 2.https://www.krea.ai/blog/krea-2-technical-report, 2026

2026

[2] [2]

Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

Pith/arXiv arXiv 2025

[3] [3]

Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.ar...

Pith/arXiv arXiv 2025

[4] [4]

All are worth words: A vit backbone for diffusion models

Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22669–22679, 2023

2023

[5] [5]

Why do llms attend to the first token?arXiv preprint arXiv:2504.02732, 2025

Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do llms attend to the first token?arXiv preprint arXiv:2504.02732, 2025

Pith/arXiv arXiv 2025

[6] [6]

Pixart-σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. InEuropean Conference on Computer Vision, pages 74–91. Springer, 2024

2024

[7] [7]

Revealing the attention floating mechanism in masked diffusion models.arXiv preprint arXiv:2601.07894, 2026

Xin Dai, Pengcheng Huang, Zhenghao Liu, Shuo Wang, Yukun Yan, Chaojun Xiao, Yu Gu, Ge Yu, and Maosong Sun. Revealing the attention floating mechanism in masked diffusion models.arXiv preprint arXiv:2601.07894, 2026

arXiv 2026

[8] [8]

Vision transformers need registers

Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In International conference on learning representations, volume 2024, pages 2632–2652, 2024

2024

[9] [9]

Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

2021

[10] [10]

Scaling rectified flow transformers for high-resolution image synthesis

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning, 2024

2024

[11] [11]

Mdtv2: Masked diffusion transformer is a strong image synthesizer.arXiv preprint arXiv:2303.14389, 2023

Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is a strong image synthesizer.arXiv preprint arXiv:2303.14389, 2023

Pith/arXiv arXiv 2023

[12] [12]

Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

2023

[13] [13]

When attention sink emerges in language models: An empirical view

Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. InInternational Conference on Learning Representations, volume 2025, pages 97114–97144, 2025

2025

[14] [14]

Conceptattention: Diffusion transformers learn highly interpretable features.arXiv preprint arXiv:2502.04320, 2025

Alec Helbling, Tuna Han Salih Meral, Ben Hoover, Pinar Yanardag, and Duen Horng Chau. Conceptattention: Diffusion transformers learn highly interpretable features.arXiv preprint arXiv:2502.04320, 2025

Pith/arXiv arXiv 2025

[15] [15]

Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

Pith/arXiv arXiv 2022

[16] [16]

Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

2020

[17] [17]

Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024. 10

Pith/arXiv arXiv 2024

[18] [18]

Diffusion transformers use sink registers

Amna Jamal, Mika Tan, Clarissa Aurelia Nahid Saputra, Quan Huynh, Kevin Zhu, and Antonio Mari. Diffusion transformers use sink registers. InSecond Workshop on XAI4Science: From Understanding Model Behavior to Discovering New Scientific Knowledge, 2025. URLhttps://openreview.net/forum?id=Fc7s3UrQr1

2025

[19] [19]

Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.Advances in Neural Information Processing Systems, 37:52481–52515, 2024

Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.Advances in Neural Information Processing Systems, 37:52481–52515, 2024

2024

[20] [20]

Vision transformers don’t need trained registers

Nicholas Jiang, Amil Dravid, Alexei Efros, and Yossi Gandelsman. Vision transformers don’t need trained registers. Advances in neural information processing systems, 38:56557–56595, 2026

2026

[21] [21]

See what you are told: Visual attention sink in large multimodal models.arXiv preprint arXiv:2503.03321, 2025

Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models.arXiv preprint arXiv:2503.03321, 2025

Pith/arXiv arXiv 2025

[22] [22]

Flux.https://github.com/black-forest-labs/flux, 2024

Black Forest Labs. Flux.https://github.com/black-forest-labs/flux, 2024

2024

[23] [23]

FLUX.2.https://bfl.ai/blog/flux-2, 2025

Black Forest Labs. FLUX.2.https://bfl.ai/blog/flux-2, 2025

2025

[24] [24]

Remarkable robustness of llms: Stages of inference? Advances in Neural Information Processing Systems, 38:130050–130083, 2026

Vedang Lad, Jin Hwa Lee, Wes Gurnee, and Max Tegmark. Remarkable robustness of llms: Stages of inference? Advances in Neural Information Processing Systems, 38:130050–130083, 2026

2026

[25] [25]

Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference.arXiv preprint arXiv:2502.20766, 2025

Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference.arXiv preprint arXiv:2502.20766, 2025

Pith/arXiv arXiv 2025

[26] [26]

Qwen-image-bench: From generation to creation in text-to-image evaluation.arXiv preprint arXiv:2605.28091, 2026

Niantong Li, Guangzheng Hu, Weixu Qiao, Ying Ba, Qichen Hong, Shijun Shen, Jinlin Wang, Fan Zhou, Jianye Kang, Xin Shang, et al. Qwen-image-bench: From generation to creation in text-to-image evaluation.arXiv preprint arXiv:2605.28091, 2026

Pith/arXiv arXiv 2026

[27] [27]

Snapkv: Llm knows what you are looking for before generation.Advances in Neural Information Processing Systems, 37:22947–22970, 2024

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation.Advances in Neural Information Processing Systems, 37:22947–22970, 2024

2024

[28] [28]

Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

Pith/arXiv arXiv 2022

[29] [29]

Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025

Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025

Pith/arXiv arXiv 2025

[30] [30]

Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time.Advances in Neural Information Processing Systems, 36:52342–52364, 2023

Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time.Advances in Neural Information Processing Systems, 36:52342–52364, 2023

2023

[31] [31]

Rethinking cross-modal interaction in multimodal diffusion transformers

Zhengyao Lv, Tianlin Pan, Chenyang Si, Zhaoxi Chen, Wangmeng Zuo, Ziwei Liu, and Kwan-Yee K Wong. Rethinking cross-modal interaction in multimodal diffusion transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 5934–5943, 2025

2025

[32] [32]

Hpsv3: Towards wide-spectrum human preference score

Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15086–15095, 2025

2025

[33] [33]

DiffSynth-Studio

ModelScope. DiffSynth-Studio. GitHub repository, 2024.https://github.com/modelscope/DiffSynth-Studio

2024

[34] [34]

Improved denoising diffusion probabilistic models

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. InInternational conference on machine learning, pages 8162–8171. PMLR, 2021

2021

[35] [35]

Scalable diffusion models with transformers

William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

2023

[36] [36]

Sdxl: Improving latent diffusion models for high-resolution image synthesis

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. InInternational Conference on Learning Representations, volume 2024, pages 1862–1874, 2024

2024

[37] [37]

Attention sinks and compression valleys in llms are two sides of the same coin.arXiv preprint arXiv:2510.06477, 2025

Enrique Queipo-de Llano, Álvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael Bronstein, Yann LeCun, and Ravid Shwartz-Ziv. Attention sinks and compression valleys in llms are two sides of the same coin.arXiv preprint arXiv:2510.06477, 2025. 11

arXiv 2025

[38] [38]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PMLR, 2021

2021

[39] [39]

Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

2020

[40] [40]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

2022

[41] [41]

Attention sinks in diffusion language models.arXiv preprint arXiv:2510.15731, 2025

Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri, Simone Scardapane, and Alessio Devoto. Attention sinks in diffusion language models.arXiv preprint arXiv:2510.15731, 2025

arXiv 2025

[42] [42]

Exploring multimodal diffusion transformers for enhanced prompt-based image editing

Joonghyuk Shin, Alchan Hwang, Yujin Kim, Daneul Kim, and Jaesik Park. Exploring multimodal diffusion transformers for enhanced prompt-based image editing. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19492–19502, 2025

2025

[43] [43]

Dinov3.arXiv preprint arXiv:2508.10104, 2025

Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025

Pith/arXiv arXiv 2025

[44] [44]

Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

Pith/arXiv arXiv 2010

[45] [45]

Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

Pith/arXiv arXiv 2011

[46] [46]

Registers matter for pixel-space diffusion transformers.arXiv preprint arXiv:2605.16147, 2026

Nikita Starodubcev, Ilia Sudakov, Ilya Drobyshevskiy, Artem Babenko, and Dmitry Baranchuk. Registers matter for pixel-space diffusion transformers.arXiv preprint arXiv:2605.16147, 2026

Pith/arXiv arXiv 2026

[47] [47]

Massive activations in large language models.arXiv preprint arXiv:2402.17762, 2024

Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models.arXiv preprint arXiv:2402.17762, 2024

Pith/arXiv arXiv 2024

[48] [48]

Razorattention: Efficient kv cache compression through retrieval heads

Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Danning Ke, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Razorattention: Efficient kv cache compression through retrieval heads. InInternational Conference on Learning Representations, volume 2025, pages 16632–16646, 2025

2025

[49] [49]

Quest: Query-aware sparsity for efficient long-context llm inference.arXiv preprint arXiv:2406.10774, 2024

Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference.arXiv preprint arXiv:2406.10774, 2024

Pith/arXiv arXiv 2024

[50] [50]

Qwen-Image-2512-Turbo-LoRA-2-Steps

Wuli Team. Qwen-Image-2512-Turbo-LoRA-2-Steps. Hugging Face model repository, 2026.https://huggingface. co/Wuli-art/Qwen-Image-2512-Turbo-LoRA-2-Steps

2026

[51] [51]

Fasa: Frequency-aware sparse attention.arXiv preprint arXiv:2602.03152, 2026

Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, Yong Wang, Ismini Lourentzou, Zhengzhong Tu, Xiangxiang Chu, and Julian McAuley. Fasa: Frequency-aware sparse attention.arXiv preprint arXiv:2602.03152, 2026

arXiv 2026

[52] [52]

Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025

Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025

Pith/arXiv arXiv 2025

[53] [53]

Attention sinks in diffusion transformers: A causal analysis.arXiv preprint arXiv:2605.09313, 2026

Fangzheng Wu and Brian Summa. Attention sinks in diffusion transformers: A causal analysis.arXiv preprint arXiv:2605.09313, 2026

Pith/arXiv arXiv 2026

[54] [54]

Efficient streaming language models with attention sinks

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InInternational Conference on Learning Representations, volume 2024, pages 21875–21895, 2024

2024

[55] [55]

Duoattention: Efficient long-context llm inference with retrieval and streaming heads

Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. InInternational Conference on Learning Representations, volume 2025, pages 37228–37253, 2025

2025

[56] [56]

Rethinking cross-layer information routing in diffusion transformers.arXiv preprint arXiv:2605.20708, 2026

Chao Xu, Maohua Li, Qirui Li, Yixuan Xu, Yanke Zhou, Yunhe Li, Cuifeng Shen, Hanlin Tang, Kan Liu, Tao Lan, et al. Rethinking cross-layer information routing in diffusion transformers.arXiv preprint arXiv:2605.20708, 2026

Pith/arXiv arXiv 2026

[57] [57]

On the nature of attention sink that shapes decoding strategy in mllms.arXiv preprint arXiv:2603.14337, 2026

Suho Yoo, Youngjoon Jang, and Joon Son Chung. On the nature of attention sink that shapes decoding strategy in mllms.arXiv preprint arXiv:2603.14337, 2026. 12

Pith/arXiv arXiv 2026

[58] [58]

The unreasonable effectiveness of deep features as a perceptual metric

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018

2018

[59] [59]

H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

2023

[60] [60]

that”,[/SYSTEM_PROMPT], and <s> as the leading sinks and “ that

Yanke Zhou, Yiduo Li, Hanlin Tang, Maohua Li, Kan Liu, Lan Tao, Lin Qu, Yuan Yao, and Xiaoxing Ma. Full attention strikes back: Transferring full attention into sparse within hundred training steps.arXiv preprint arXiv:2605.16928, 2026. 13 Appendix Causal Interpretability Framework for DiTs MM-ATTN Block Head Head Head Head Block Sink Heads 01 02 03 04 I2...

Pith/arXiv arXiv 2026