Pith. sign in

REVIEW 4 major objections 5 minor 58 references

Sparse Fine-Tuning of Transformers for Generative Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning a transformer can be recast as sparse coding, with the adapted feature at each layer a sparse combination of learned feature-dictionary atoms that serve as the interpretable units of adaptation.

desk verdict A clean sparse-dictionary parameterization for fine-tuning transformers, worth reviewing, but the DreamBooth evaluation is under-powered and the editing claims are qualitative. read the letter →

arxiv 2507.10855 v1 pith:JS76CPLV submitted 2025-07-14 cs.CV

Wei Chen , Jingxi Yu , Zichen Miao , Qiang Qiu This is my paper
classification cs.CV
keywords sparsefine-tuningcodingfeaturedictionaryinterpretabilityimageeditingconceptcustomizationdiffusiontransformersparameter-efficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that fine-tuning a pre-trained transformer for a generative task can be reformulated as sparse coding: the adaptation at every attention layer is a sparse linear combination of learned “feature dictionary atoms,” not a dense change to weights. The claimed payoff is interpretability and control—sparse coefficients indicate which atoms matter, so dropping unimportant atoms improves text-aligned image editing, and building a new concept from a few atoms works for personalization. The paper supports this with a toy signal-reconstruction experiment showing that tuning the atoms (not the coefficients) carries adaptation, and with customization experiments on a diffusion-transformer text-to-image model. If the claim is right, fine-tuned generative models can be understood and edited at the level of individual semantic building blocks rather than as opaque weight updates.

What carries the argument

The load-bearing object is the sparse dictionary factorization $\Delta O = \sigma_\lambda(A X W_s) D$ imposed on the adapted feature at each attention layer: $D \in \mathbb{R}^{M \times C_o}$ is the learned feature dictionary (the atoms), $W_s$ produces the coefficient pre-activations, and $\sigma_\lambda$ enforces sparsity so that only a few atoms contribute to each output. This factorization turns fine-tuning into dictionary learning—freeze the pre-trained output $O$, learn $D$ and $W_s$ for the downstream task—and sparsity is what makes the coefficients usable as atom-importance indicators. A supplementary polynomial-expansion analysis is used to argue that shallow-layer atoms shape global structure while deep-layer atoms contribute detail.

What would settle it

On a large set of personalization subjects, measure at each attention layer the reconstruction error of the observed adapted feature $\Delta O$ against its best sparse approximation using $M=256$ atoms; if many subjects require far more than 4% active atoms to reach the fidelity of the full fine-tuned model, the sparse-span premise fails. A complementary test: with the learned dictionary $D$ fixed, replace the sparse coefficients by dense low-rank coefficients of equal parameter count and compare prompt-modified generations; if the dense version is equally stable, sparsity itself is not what the experiments demonstrate.

Watch

Extended reading notes

Core claim

The central claim is that the residual feature that fine-tuning needs to add at each attention layer can be written as $\Delta O = \sigma_\lambda(A X W_s) D$, where $D$ is a learned dictionary of $M$ feature atoms, $A$ is the attention map, $X$ the input, $W_s$ a learned projection, and $\sigma_\lambda$ a sparsifying nonlinearity such as soft-thresholding, shifted ReLU, or top-k activation. The adapted output is $O + \Delta O$, with $O$ from frozen pre-trained weights, so the pre-trained model is untouched and the new knowledge is a sparse combination of atoms. The paper argues that the sparse coefficients are importance indicators, that the atoms themselves are the units that must be tuned, and that a small number of atoms suffices: a toy Fourier experiment shows atoms (not coefficients) carry the new task, a two-layer transformer VAE flips generated “8” digits into “3” digits with 12 atoms, and a diffusion-transformer image model reconstructs a target concept with about 40 atoms, with 4 atoms fixing its core structure.

Load-bearing premise

The method assumes that the small adjustment a fine-tuned model needs at every attention layer can be expressed as a sparse combination of a few hundred learned dictionary atoms—with only 2–4% of atoms active—and that this sparse span is rich enough for arbitrary generative concepts; the experiments test this on a limited set of subjects and mostly qualitative editing examples.

Editorial extensions

If this is right

  • A fine-tuned concept can be represented by a small set of dictionary atoms (about 40 in the paper's experiments, with the core structure in about 4), so adaptation is sparse and inspectable rather than a dense weight change.
  • Editing an image becomes a matter of choosing coefficient density: raising sparsity keeps only the most essential atoms and improves text alignment, while lowering sparsity adds fidelity.
  • Sparse coefficient updates are more stable than low-rank ones under prompt changes: with a single active atom the method keeps generating the intended concept under a modified prompt, whereas the rank-one low-rank baseline produces disrupted outputs.
  • On the 30-subject personalization benchmark used in the paper, the sparse dictionary method improves text-to-image alignment and diversity over the compared fine-tuning baselines while preserving comparable fidelity.
  • The dictionary size $M$ is a controllable trade-off: a larger dictionary improves fidelity, while a smaller dictionary improves alignment and diversity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sparse dictionary view holds across layers, the same atoms could be clustered by the visual attributes they control (texture, pose, object parts), which would turn fine-tuning into a direct concept-editing interface; the paper only shows this qualitatively with a few examples.
  • The toy experiment's conclusion—that tuning atoms matters more than tuning coefficients—predicts a direct ablation on the text-to-image model: fine-tuning only value/output projections should outperform fine-tuning only query/key projections at matched capacity. That comparison is not reported.
  • The expressivity bet that a few hundred atoms span any needed residual could be tested systematically by measuring how reconstruction error of $\Delta O$ drops as $M$ grows across many subjects; the paper reports the effect of $M$ on generation metrics but not the underlying reconstruction error.
  • A stronger falsification of the sparsity mechanism would replace the sparse coefficients with dense low-rank coefficients of the same parameter count while keeping the same atoms; if stability under prompt changes persists, the benefit comes from the dictionary parameterization rather than from sparsity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a sparse-coding view of fine-tuning for transformers in generative tasks. Instead of adapting weight matrices densely, the method represents the adapted feature residual ΔO at each attention layer as ΔO = σλ(AXWs)D, where D is a learned dictionary of feature atoms and σλ is a sparsifying nonlinearity (soft-threshold, ReLU, or top-k). The authors argue that this sparse representation makes fine-grained adaptation interpretable and controllable. They demonstrate the formulation on a 1D Fourier toy task, a 2-layer VAEs digit-generation experiment, qualitative image editing on DiT (PixArt-Σ), and DreamBooth concept customization comparing against LoRA, DoRA, and OFT. The central claim is that sparse dictionary atoms act as building blocks for new concepts, enabling better text-aligned image editing and concept customization than existing dense or low-rank fine-tuning methods.

Significance. If the central claim is established, the paper offers a genuinely different perspective on fine-tuning: instead of modifying weights or dense residual features, it represents the adapted representation as a sparse combination of a small dictionary, which could improve interpretability and controllability of generative transformers. The formulation in Eq. (8)-(9) is simple and clean, and the analysis in Section 3.3 distinguishing the roles of atoms versus coefficients is a useful conceptual contribution. The paper also provides a constructive parameter-count and FLOPs comparison with LoRA in Section 3.5. However, the empirical evidence is currently too weak to support the headline claims: the main quantitative comparison (Table 1) lacks error bars and uses post hoc density selection, and the editing results are entirely qualitative. The interpretability claims rely partly on definitional properties of the parameterization and on an unverified orthogonality assumption in the influence decomposition. With additional experiments and analysis, the ideas could be a solid contribution, but in its present form the significance is not fully demonstrated.

major comments (4)
  1. [Section 4.3, Table 1] The claim that the proposed method outperforms LoRA, DoRA, and OFT on concept customization is not supported by the evidence as reported. Table 1 presents a single run per method with no standard errors, confidence intervals, or significance tests, so the reader cannot assess the stability of the reported differences. Moreover, the two densities ρ = 0.02 and ρ = 0.04 are selected post hoc per metric: Section 4.4 states that the authors adopt both 'for either better diversity or better fidelity'. For any metric, there are two chances to beat the baselines, which inflates the apparent advantage. Please report multiple seeds with error bars or a fixed model-selection rule (e.g., choose ρ on a validation set and keep it fixed for all metrics), and state which density is used for each reported number.
  2. [Section 4.2, Figures 5-6] The image-editing claim is currently qualitative. There is no quantitative evaluation, no comparison with editing-specific baselines (e.g., SINE [58] or another standard editing method), and no ablation that isolates atom removal from simply increasing sparsity of the residual. As a result, the proposed mechanism—that removing unimportant atoms improves text alignment—is not actually tested; the observed differences in Figures 5-6 could come from stronger sparsity alone or from better hyperparameters. Please add a quantitative editing benchmark, or at least compare against a sparsity-only variant of the same fine-tuning objective under identical training budgets.
  3. [Section 3.2, Eq. (9); Section 4.1, Figure 3] The claim that atoms are semantic building blocks is partly definitional: ΔO is, by construction, a sparse combination of the learned atoms, so the fact that a few atoms can reconstruct a concept when the dictionary is trained per concept on a single image is not independent evidence of interpretability. The per-concept dictionary can overfit the single image, and the observation that 4 atoms approximate the concept may reflect the dictionary's capacity rather than any semantic decomposition. Without testing whether atoms transfer across concepts or whether the same dictionary structure recovers interpretable parts when trained on multiple concepts, the interpretability claim is not established. This matters because it is the basis for the proposed editing application.
  4. [Appendix A, Eqs. (11)-(13); Figure 1(c)] The influence decomposition assumes that dictionary atoms are mutually orthogonal, ⟨di,dj⟩=0 for i≠j. The paper states that this 'can be achieved with a simple regularization term', but no such regularizer is used or reported in any experiment. Therefore the atom-importance visualizations in Figure 1(c) and the associated claims about which atoms contribute to the generated digit rely on an unverified assumption. Please either add the orthogonality regularizer and verify the assumption, or report the decomposition with the actual non-orthogonal atoms and quantify the resulting approximation error.
minor comments (5)
  1. [Section 4.4 heading] The heading 'Abalation Study' contains a typo; it should be 'Ablation Study'.
  2. [Section 3.5 and Section 4.2] There are typos: 'compututation' in Section 3.5 and 'addtion' in Section 4.2.
  3. [Figure ordering] Figure 4 is referenced and discussed in Section 3.5, before Figure 3 is introduced in Section 4.1. The figures should be reordered or renumbered so that they appear in order of first reference.
  4. [Section 3.5 vs Section 4.3] The parameter-count example in Section 3.5 reports 264k parameters for the proposed method versus 131k for LoRA, while Section 4.3 reports approximately 17M versus 4M for the full model. These numbers are not contradictory because the Section 3.5 example assumes a particular configuration (M=256, Ci=Co=1024), but the discrepancy could confuse readers. Please clarify that the two sets of numbers correspond to different configurations.
  5. [Section 4.3 and Appendix B] The main text says the concept customization experiment uses the AdamW optimizer with learning rate 1e-4, while Appendix B states the CAME optimizer with learning rate 1e-4 for the DiT experiments. Please reconcile these descriptions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the sparse-atom formulation is a modeling choice, and the core evidence comes from external benchmarks and controlled toy experiments.

full rationale

The paper's key equation (9) defines the adapted feature as O + Delta-O = O + sigma_lambda(A X W_s) D. Statements such as 'feature dictionary atoms function as fundamental building blocks' are descriptions of that parameterization rather than predictions derived from it. The empirical claims--customization metrics in Table 1, editing results in Figures 5-6, the MNIST atom-influence study, and the Fourier toy experiment in Section 3.3--are evaluated against external quantities: LoRA/DoRA/OFT baselines, CLIP/ImageReward scores, and reconstruction of synthetic signals. The sparse coefficients S are learned from data, but the paper does not fit a parameter to a subset of data and then 'predict' that same subset; the density rho is a hyperparameter chosen post hoc, which is a statistical robustness issue, not circularity. The related-work self-citations ([8], [33]) are not load-bearing: no uniqueness theorem or crucial ansatz is imported from them to force the sparse dictionary form. The definitional framing could be called tautological in the sense that Delta-O is by construction a sparse combination of atoms, but no derived result reduces to its own input, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the expressivity of a sparse dictionary per layer and on unenforced orthogonality for the interpretability analysis. No new physical entities are introduced. The free parameters are hyperparameters chosen by hand or post hoc, which weakens the empirical case.

free parameters (4)
  • Sparsity threshold λ = not reported
    Used in σλ (soft-thresholding or ReLU) to control sparsity; no value or sensitivity analysis given.
  • Coefficient density ρ = 0.01, 0.02, 0.04 (selected per experiment)
    Controls fraction of active atoms; chosen post hoc to balance diversity and fidelity (Section 4.4).
  • Dictionary size M = 256 (main), 100 (MNIST), 16-2048 (ablation)
    Number of atoms per layer; M=256 chosen for balanced fidelity and diversity (Table 2).
  • Number of active atoms for editing visualization = 4, 40, approximately all
    In Section 4.1, atom counts are manually varied to demonstrate controllability.
assumptions (4)
  • domain assumption Residual adapted features at each attention layer lie in a sparse linear span of a small learned dictionary D, with coefficients σλ(AXWs).
    This is the core expressivity assumption behind Eq. (8)-(9); no proof or capacity analysis is given.
  • domain assumption Freezing pre-trained weights and adding a residual dictionary preserves the pre-trained model's capabilities.
    Assumed throughout; experiments suggest no collapse but there is no systematic measurement of capability preservation.
  • ad hoc to paper Atoms are mutually orthogonal for the influence decomposition in Appendix A.
    The paper states this 'can be achieved with a simple regularization term' but does not report using such a term in the main experiments.
  • domain assumption Findings from a 1D Fourier signal toy experiment transfer to large diffusion transformers.
    Section 3.3 uses the toy to conclude that 'tuning atoms is more suitable than tuning coefficients', but the real method trains both W_s and D, and the toy domain is far from text-to-image generation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sparse Fine-Tuning of Transformers for Generative Tasks." pith.science (2026). https://pith.science/paper/JS76CPLV

@misc{pith2026250710855,
  author       = {Pith},
  title        = {Pith review of: Sparse Fine-Tuning of Transformers for Generative Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JS76CPLV}},
  note         = {Machine review of arXiv:2507.10855}
}
read the original abstract

Large pre-trained transformers have revolutionized artificial intelligence across various domains, and fine-tuning remains the dominant approach for adapting these models to downstream tasks due to the cost of training from scratch. However, in existing fine-tuning methods, the updated representations are formed as a dense combination of modified parameters, making it challenging to interpret their contributions and understand how the model adapts to new tasks. In this work, we introduce a fine-tuning framework inspired by sparse coding, where fine-tuned features are represented as a sparse combination of basic elements, i.e., feature dictionary atoms. The feature dictionary atoms function as fundamental building blocks of the representation, and tuning atoms allows for seamless adaptation to downstream tasks. Sparse coefficients then serve as indicators of atom importance, identifying the contribution of each atom to the updated representation. Leveraging the atom selection capability of sparse coefficients, we first demonstrate that our method enhances image editing performance by improving text alignment through the removal of unimportant feature dictionary atoms. Additionally, we validate the effectiveness of our approach in the text-to-image concept customization task, where our method efficiently constructs the target concept using a sparse combination of feature dictionary atoms, outperforming various baseline fine-tuning methods.

Figures

Figures reproduced from arXiv: 2507.10855 by the authors.

Figure 1
Figure 1. (a) Our method leverages sparse coding and enforces high sparsity in adapted feature [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Adjusting only the coefficients fails to adapt the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The pre-trained model can generate the target concept when integrated with the learned dictionary, even with about 40 atoms. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The role of sparsity: Our method utilizes the same num [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Adjusting the density, i.e., the number of activated atoms, can change the influence on the edited image. 4.3. Concept Customization In the previous section, we demonstrated that our method can efficiently represent the target concept using only a small number of atoms…
Figure 5
Figure 5. Figure 5: Results on image editing. Our method shows improved [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: The influence of different density ρ on the image diver￾sity, fidelity and text-to-image alignment. method achieves higher text-to-image alignment and di￾versity scores than most other PEFT baselines, while still preserving a relatively high fidelity score. This indica…
Figure 8
Figure 8. Figure 8: The illustration of Fourier basis. Generative tasks with DiT. In this experiment, we use the CAME [26] optimizer with a learning rate of 1×10−4 to fine-tune the Pixart-Σ [7]. For the baseline methods, LoRA and DoRA are assigned a rank of r = 16, while OFT is set to r =…
Figure 9
Figure 9. Figure 9: Compared to the performance of (a) standard atten [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Personalization generated results comparison [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Personalization generated results comparison [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Personalization generated results comparison [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Personalization generated results comparison [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Personalization generated results comparison [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Personalization generated results comparison [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 45 canonical work pages

  1. [58]

    A grey ⟨V ⟩ wolf plushie

    Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris N Metaxas, and Jian Ren. Sine: Single image editing with text- to-image diffusion models. In CVPR, 2023. 6, 7 Sparse Fine-Tuning of Transformers for Generative Tasks Supplementary Material A. Analysis A.1. Analysis of Adapted Feature Representation With our formulation, the adapted feature representation ∆O...

    work page 2023

  2. [1]

    Decomposing and interpreting image representations via text in vits beyond CLIP

    Sriram Balasubramanian, Samyadeep Basu, and Soheil Feizi. Decomposing and interpreting image representations via text in vits beyond CLIP. In The Thirty-eighth An- nual Conference on Neural Information Processing Systems,

    work page

  3. [2]

    A fast iterative shrinkage- thresholding algorithm for linear inverse problems

    Amir Beck and Marc Teboulle. A fast iterative shrinkage- thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2009. 3

    work page 2009

  4. [3]

    Long- former: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. 2

    arXiv 2004

  5. [4]

    Rep- resentation learning: A review and new perspectives

    Yoshua Bengio, Aaron Courville, and Pascal Vincent. Rep- resentation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798–1828, 2013. 5

    work page 2013

  6. [5]

    Towards monosemanticity: De- composing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yi- fan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. To...

    work page 2023

  7. [6]

    Compressive sampling

    Emmanuel J Cand `es et al. Compressive sampling. In Pro- ceedings of the international congress of mathematicians , pages 1433–1452, 2006. 3

    work page 2006

  8. [7]

    Pixart- σ: Weak-to-strong training of dif- fusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- σ: Weak-to-strong training of dif- fusion transformer for 4k text-to-image generation. In Euro- pean Conference on Computer Vision, pages 74–91, 2024. 6, 7, 2

    work page 2024

Show all 58 references
  1. [8]

    Large convolutional model tuning via filter subspace

    Wei Chen, Zichen Miao, and Qiang Qiu. Large convolutional model tuning via filter subspace. International Conference on Learning Representations, 2025. 2

    2025

  2. [9]

    Graph con- volution with low-rank learnable local filters

    Xiuyuan Cheng, Zichen Miao, and Qiang Qiu. Graph con- volution with low-rank learnable local filters. arXiv preprint arXiv:2008.01818, 2020. 1

    2008 arXiv

  3. [10]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. 2

    1904 arXiv

  4. [11]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Trans- formers for image recognition at scale. ICLR, 2021. 1

    2021

  5. [12]

    Transcoders enable fine-grained interpretable circuit analy- sis for language models

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders enable fine-grained interpretable circuit analy- sis for language models. 2024. AI Alignment Forum. 2

    2024

  6. [13]

    The vendi score: A diversity evaluation metric for machine learning

    Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. arXiv preprint arXiv:2210.02410, 2022. 7

    2022 arXiv

  7. [14]

    Svdiff: Compact param- eter space for diffusion fine-tuning

    Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact param- eter space for diffusion fine-tuning. In CVPR, 2023. 7

    2023

  8. [15]

    Conceptexpress: Harnessing diffusion models for single-image unsupervised concept extraction

    Shaozhe Hao, Kai Han, Zhengyao Lv, Shihao Zhao, and Kwan-Yee K Wong. Conceptexpress: Harnessing diffusion models for single-image unsupervised concept extraction. In European Conference on Computer Vision, 2024. 2

    2024

  9. [16]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. In International Conference on Learning Representations, 2021. 1, 7, 8

    2021

  10. [17]

    Sara: High-efficient diffusion model fine-tuning with progressive sparse low-rank adapta- tion

    Teng Hu, Jiangning Zhang, Ran Yi, Hongrui Huang, Yabiao Wang, and Lizhuang Ma. Sara: High-efficient diffusion model fine-tuning with progressive sparse low-rank adapta- tion. arXiv preprint arXiv:2409.06633, 2024. 2

    2024 arXiv

  11. [18]

    Text embed- ding is not all you need: Attention control for text-to-image semantic alignment with text self-attention maps

    Jeeyung Kim, Erfan Esmaeili, and Qiang Qiu. Text embed- ding is not all you need: Attention control for text-to-image semantic alignment with text self-attention maps. In Pro- ceedings of the Computer Vision and Pattern Recognition Conference, pages 8031–8040, 2025. 2

    2025

  12. [19]

    Learning to cus- tomize text-to-image diffusion in diverse context

    Taewook Kim, Wei Chen, and Qiang Qiu. Learning to cus- tomize text-to-image diffusion in diverse context. arXiv preprint arXiv:2410.10058, 2024. 2

    2024 arXiv

  13. [20]

    An introduction to variational autoencoders

    Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends® in Ma- chine Learning, 2019. 5

    2019

  14. [21]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 1

    2023

  15. [22]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,

  16. [23]

    Flexprefill: A context-aware sparse attention mech- anism for efficient long-sequence inference

    Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context-aware sparse attention mech- anism for efficient long-sequence inference. arXiv preprint arXiv:2502.20766, 2025. 2

    2025 arXiv

  17. [24]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024. 1, 7, 8

    2024 arXiv

  18. [25]

    Re-imagining multimodal instruction tuning: A representation view

    Yiyang Liu, James Chenhao Liang, Ruixiang Tang, Yugyung Lee, Majid Rabbani, Sohail Dianat, Raghuveer Rao, Lifu Huang, Dongfang Liu, Qifan Wang, et al. Re-imagining multimodal instruction tuning: A representation view. arXiv preprint arXiv:2503.00723, 2025. 2

    2025 arXiv

  19. [26]

    Came: Confidence-guided adaptive memory efficient optimization

    Yang Luo, Xiaozhe Ren, Zangwei Zheng, Zhuo Jiang, Xin Jiang, and Yang You. Came: Confidence-guided adaptive memory efficient optimization. arXiv preprint arXiv:2307.02047, 2023. 2

    2023 arXiv

  20. [27]

    Supervised dictionary learning

    Julien Mairal, Jean Ponce, Guillermo Sapiro, Andrew Zis- serman, and Francis Bach. Supervised dictionary learning. Advances in neural information processing systems, 2008. 1

    2008

  21. [28]

    Sparse feature cir- cuits: Discovering and editing interpretable causal graphs in language models

    Samuel Marks, Can Rager, Eric J Michaud, Yonatan Be- linkov, David Bau, and Aaron Mueller. Sparse feature cir- cuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647, 2024. 2

    2024 arXiv

  22. [29]

    Con- tinual learning with filter atom swapping

    Zichen Miao, Ze Wang, Wei Chen, and Qiang Qiu. Con- tinual learning with filter atom swapping. In International Conference on Learning Representations, 2021. 1

    2021

  23. [30]

    Spatiotemporal joint filter decomposition in 3d convolutional neural networks

    Zichen Miao, Ze Wang, Xiuyuan Cheng, and Qiang Qiu. Spatiotemporal joint filter decomposition in 3d convolutional neural networks. Advances in Neural Information Process- ing Systems, 34:3376–3388, 2021. 1

    2021

  24. [31]

    Training diffusion models towards diverse image generation with reinforcement learning

    Zichen Miao, Jiang Wang, Ze Wang, Zhengyuan Yang, Li- juan Wang, Qiang Qiu, and Zicheng Liu. Training diffusion models towards diverse image generation with reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 108...

    2024

  25. [32]

    Tuning timestep- distilled diffusion model using pairwise sample optimiza- tion

    Zichen Miao, Zhengyuan Yang, Kevin Lin, Ze Wang, Zicheng Liu, Lijuan Wang, and Qiang Qiu. Tuning timestep- distilled diffusion model using pairwise sample optimiza- tion. arXiv preprint arXiv:2410.03190, 2024. 2

    2024 arXiv

  26. [33]

    Coeff-tuning: A graph filter subspace view for tuning attention-based large models

    Zichen Miao, Wei Chen, and Qiang Qiu. Coeff-tuning: A graph filter subspace view for tuning attention-based large models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20146–20157, 2025. 2

    2025

  27. [34]

    Emergence of simple- cell receptive field properties by learning a sparse code for natural images

    Bruno A Olshausen and David J Field. Emergence of simple- cell receptive field properties by learning a sparse code for natural images. Nature, pages 607–609, 1996. 1

    1996

  28. [35]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 7

    2023 arXiv

  29. [36]

    Ef- ficient source-free time-series adaptation via parameter sub- space disentanglement

    Gaurav Patel, Christopher Sandino, Behrooz Mahasseni, Ellen L Zippi, Erdrin Azemi, Ali Moin, and Juri Minxha. Ef- ficient source-free time-series adaptation via parameter sub- space disentanglement. arXiv preprint arXiv:2410.02147 ,

    arXiv

  30. [37]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In CVPR, 2023. 6

    2023

  31. [38]

    Posterior sampling via langevin dynamics based on generative priors.arXiv preprint arXiv:2410.02078, 2024

    Vishal Purohit, Matthew Repasky, Jianfeng Lu, Qiang Qiu, Yao Xie, and Xiuyuan Cheng. Posterior sampling via langevin dynamics based on generative priors.arXiv preprint arXiv:2410.02078, 2024. 2

    2024 arXiv

  32. [39]

    Dcfnet: Deep neural network with decomposed convolutional filters

    Qiang Qiu, Xiuyuan Cheng, Guillermo Sapiro, et al. Dcfnet: Deep neural network with decomposed convolutional filters. In International Conference on Machine Learning , pages 4198–4207. PMLR, 2018. 1

    2018

  33. [40]

    Controlling text-to-image diffusion by orthogo- nal finetuning

    Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Sch¨olkopf. Controlling text-to-image diffusion by orthogo- nal finetuning. Advances in Neural Information Processing Systems, 36:79320–79362, 2023. 1, 7, 8

    2023

  34. [41]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, 2021. 7

    2021

  35. [42]

    Unveiling and mitigating mem- orization in text-to-image diffusion models through cross at- tention

    Jie Ren, Yaxin Li, Shenglai Zeng, Han Xu, Lingjuan Lyu, Yue Xing, and Jiliang Tang. Unveiling and mitigating mem- orization in text-to-image diffusion models through cross at- tention. In European Conference on Computer Vision, 2024. 2

    2024

  36. [43]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, pages 10684– 10695, 2022. 1

    2022

  37. [44]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  38. [45]

    Unpacking sdxl turbo: Interpreting text-to-image models with sparse au- toencoders

    Viacheslav Surkov, Chris Wendler, Mikhail Terekhov, Justin Deschenaux, Robert West, and Caglar Gulcehre. Unpacking sdxl turbo: Interpreting text-to-image models with sparse au- toencoders. arXiv preprint arXiv:2410.22366, 2024. 2

  39. [46]

    Sparse sinkhorn attention

    Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da- Cheng Juan. Sparse sinkhorn attention. In International conference on machine learning, 2020. 2

    2020

  40. [47]

    Longer attention span: Increasing transformer context length with sparse graph processing techniques

    Nathaniel Tomczak and Sanmukh Kuppannagari. Longer attention span: Increasing transformer context length with sparse graph processing techniques. arXiv preprint arXiv:2502.01659, 2025. 2

    2025 arXiv

  41. [48]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 2017. 1, 2, 3

    2017

  42. [49]

    Stochastic conditional generative networks with basis de- composition

    Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis de- composition. arXiv preprint arXiv:1909.11286, 2019. 1

    1909 arXiv

  43. [50]

    Image generation using continuous filter atoms

    Ze Wang, Seunghyun Hwang, Zichen Miao, and Qiang Qiu. Image generation using continuous filter atoms. Advances in Neural Information Processing Systems, 34:17826–17838, 2021

    2021

  44. [51]

    Adaptive convolutions with per-pixel dynamic filter atom

    Ze Wang, Zichen Miao, Jun Hu, and Qiang Qiu. Adaptive convolutions with per-pixel dynamic filter atom. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 12302–12311, 2021. 1

    2021

  45. [52]

    Advancing parameter efficiency in fine-tuning via representation editing

    Muling Wu, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xi- aoqing Zheng, and Xuanjing Huang. Advancing parameter efficiency in fine-tuning via representation editing. 2024. 2

    2024

  46. [53]

    Reft: Representation finetuning for language models

    Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christo- pher Potts. Reft: Representation finetuning for language models. 2024. 2

    2024

  47. [54]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation. Advances in Neural Information Pro- cessing Systems, 2023. 7

    2023

  48. [55]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y . X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse atten- tion: Hardware-aligned and natively trainable sparse atten-...

    2025

  49. [56]

    Enhancing semantic fidelity in text-to-image synthesis: Attention regulation in diffusion models

    Yang Zhang, Teoh Tze Tzun, Lim Wei Hern, and Kenji Kawaguchi. Enhancing semantic fidelity in text-to-image synthesis: Attention regulation in diffusion models. In Euro- pean Conference on Computer Vision, 2024. 2

    2024

  50. [57]

    Object- conditioned energy-based attention map alignment in text-to- image diffusion models

    Yasi Zhang, Peiyu Yu, and Ying Nian Wu. Object- conditioned energy-based attention map alignment in text-to- image diffusion models. In European Conference on Com- puter Vision, 2024. 2

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.