Pith. sign in

REVIEW 6 major objections 6 minor 56 references

MixSA: Training-free Reference-based Sketch Extraction via Mixture-of-Self-Attention

T0 review · 6 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By swapping self-attention keys and values inside a pretrained diffusion U-Net, this paper claims, sketches can be extracted from photos in any reference brushstroke style without any training.

desk verdict A useful training-free sketch extraction method with convincing qualitative results, but the ζ control axis is described backwards in the text, and the quantitative and theoretical support is thinner than claimed. read the letter →

arxiv 2501.00816 v1 pith:WK6HSY3M submitted 2025-01-01 cs.CV

Rui Yang , Xiaojun Wu , Shengfeng He This is my paper · ORCID
classification cs.CV
keywords sketchextractionreference-basedstyleself-attentionmanipulationdiffusionmodelsDDIMinversiontraining-freeimage-to-imagetranslationstabletexturedensitycontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that sketch extraction from a color photo can be done with no training while faithfully copying the brushstroke style of an arbitrary reference sketch. The proposed method, MixSA, starts from the photo's edge map as the initial latent, then during denoising replaces the self-attention keys and values in late decoder layers of Stable Diffusion with those of the reference sketch, and mixes the query features from photo, edge map, and reference with two user-controlled weights. A binarization post-step counters Stable Diffusion's tendency to average colors toward mid-gray, preserving crisp black-and-white lines. If the claim holds, artists could generate sketches in any style, or blended styles, from a single reference image and a photo, without paired training data.

What carries the argument

The key mechanism is the mixture-of-self-attention operation, written as $\phi^s_{\text{out}} = \text{Attn}(Q^m_t, K^r_t, V^r_t)$, where $K^r_t$ and $V^r_t$ are the reference sketch's self-attention keys and values taken from its DDIM-inverted denoising trajectory, and $Q^m_t = \zeta(\beta Q^c_t + (1-\beta)Q^s_t) + (1-\zeta)Q^r_t$ is a query fused from the colored image's query, the edge map's query, and the reference's query. This operation is the single point where brushstroke texture enters the generation; the Decomposing Contours and Texture (DCT) module supplies the $\beta$-weighted query mixture, the initial-outline starting point supplies structure, and the binarization step supplies color correction. It is applied only in late decoder self-attention layers, which the paper identifies as the layers responsible for local textures.

What would settle it

Encode a color photo and its edge map separately with DDIM inversion, form the latent mixture $\beta z_{\text{color}} + (1-\beta)z_{\text{edge}}$ for several $\beta$ values, decode each mixture back to an image, and compare against the same convex combination done directly in pixel space; if the decoded mixtures do not track the pixel-space blends monotonically, the additive latent-space assumption underlying Eq. (5) is violated.

Watch

Extended reading notes

Core claim

The central discovery the author is trying to establish is that reference brushstroke style can be transferred into a sketch purely by editing self-attention inside the denoising U-Net of a pretrained latent diffusion model. Concretely, the reference sketch's keys and values, captured via DDIM inversion, replace those of the sketch being generated, while the query is a weighted sum of the color photo's query, the edge map's query, and the reference's query, with weights $\beta$ and $\zeta$. The paper argues that late decoder self-attention layers around layers 10 and 11 handle local stroke textures, so injecting the reference's keys and values there is what aligns brushstroke style with the photo's contours. It further claims that using the photo's edge map as the starting latent, together with contrast-enhancing binarization of extreme brightness pixels, fixes the color-averaging artifact that otherwise washes out black-and-white sketches.

Load-bearing premise

The load-bearing premise is that in the latent space of Stable Diffusion, the features of a colored image and an edge map can be linearly added and subtracted, so the $\beta$-weighted query mixture behaves like a coherent blend of photo texture and edge contours; the paper states this assumption explicitly in Section IV.B.

Editorial extensions

If this is right

  • Users can extract a sketch in any reference style, including styles absent from any training set, by supplying one photo and one reference sketch at inference time.
  • The user dials $\zeta$ and $\beta$ give explicit control over reference adherence and texture retention, so the same pipeline spans sparse, abstract strokes to dense, detailed ones.
  • Blending two reference sketches by adjusting the query mixture yields intermediate, previously unseen sketch styles, as demonstrated with Xieyi and Gongbi styles.
  • Because no training is needed, switching to a different reference style is instantaneous at inference, unlike the paired-data baselines Ref2sketch and Semi-ref2sketch.
  • With foreground extraction enabled, the method can sketch only the salient subject, ignoring a distracting background.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same key/value substitution likely transfers other line-based visual attributes, such as hatching density or pen pressure variation, that the paper's style set does not explicitly test; this is my inference, not a claim of the paper.
  • The latent linearity assumption could be probed directly on the autoencoder latents, and if it holds only approximately, the $\zeta$ and $\beta$ dials might be re-parameterized as image-domain attention-mask interpolation.
  • Since the mechanism is backbone-agnostic, the same recipe should transfer to newer latent diffusion models by identifying the analogous late-decoder self-attention layers, though the paper only tests Stable Diffusion 1.4.
  • In practice, setting $\zeta$ and $\beta$ per style may require a small manual search; a learned or automatic parameter selector would be a natural follow-up, and is not addressed in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The manuscript introduces MixSA, a training-free method for extracting a sketch from a color photograph in the style of a user-supplied reference sketch. The pipeline uses DDIM inversion to obtain latents for the color image, a TEED edge map, and the reference sketch; during denoising it replaces the self-attention keys and values of the edge-map stream with those of the reference, and forms the queries as a weighted combination of color, edge, and reference queries (Eqs. 5-7). The paper claims precise control over texture density and reference adherence via parameters ζ and β, interpolation between styles to create unseen styles, and a solution to the 'color averaging' artifact of Stable Diffusion. Quantitative comparisons are reported on 4SKST, FS2K, and Anime, together with an ablation study and a user study.

Significance. MixSA addresses a real limitation: prior sketch-extraction models are tied to training styles, while prior training-free style transfer does not preserve line and contour structure. The attention-swapping mechanism is simple, plausible, and supported by the qualitative figures: the reference brushstroke structure appears in the outputs, and the ζ/β axes visibly change texture density and reference adherence. The user study (71.68%) is suggestive, and the FS2K ablation shows that each component contributes. These strengths make the method a reasonable candidate for TVCG. However, as written the paper contains several load-bearing problems: a direct contradiction between Eq. (6) and the text/figures for the central control parameter ζ; an incorrect frequency-domain justification for the color-averaging analysis; a headline 'superior performance' claim contradicted by the FID result in Table I; a parameter-tuning protocol that does not establish generalization; and an unvalidated linearity assumption at the core of the query-mixing formula. These issues prevent acceptance in the current form.

major comments (6)
  1. [IV.C.1 (Eq. 6), IV.C.3, Fig. 5] The semantics of ζ are stated in two opposite directions. Eq. (6) defines Qm_t = ζ Qcs_t + (1−ζ) Qr_t, so ζ=1 gives the pure contour/color query and ζ=0 gives the pure reference query. The text immediately after Eq. (6) says 'A higher ζ value results in greater preservation of Qr_t', Section IV.C.3 repeats that higher ζ enhances preservation of the reference brushstroke styles, and the caption of Fig. 5 says increasing ζ makes strokes approach the reference sketch. These statements would be correct for (1−ζ), not ζ, and the same reversal appears in the supplementary Fig. A2 caption. Because ζ is presented as the paper's precise control and interpolation axis, the method is not reproducible as written. Please either change Eq. (6) to Qm_t = (1−ζ) Qcs_t + ζ Qr_t or revise all textual descriptions and figure captions, and then re-examine Fig. 7 and the supplementary parameter plots accordingly.
  2. [IV.D, Eqs. (8)-(10)] The frequency-domain argument is not correct. In Eq. (8), the noise ε is i.i.d. Gaussian; under any orthonormal frequency decomposition the noise variance is the same for each frequency component, so the cumulative variances in Eq. (9) are equal, not related by ≫. The correct intuition is that natural images have most of their signal power at low frequencies, so high-frequency components have a lower signal-to-noise ratio and are harder to reconstruct; it is not that high-frequency components accumulate more noise variance. Since this subsection is used to justify the initial-contour anchoring and binarization (the third claimed contribution), the argument should be corrected or reframed.
  3. [V.D, Table I; Abstract] The abstract and introduction claim 'superior performance' and results that surpass other methods 'in both quality and fidelity', but in Table I MixSA's FID on 4SKST is 89.04, worse than Semi-ref2sketch's 82.18. FID is one of the paper's headline metrics, so the quantitative superiority claim needs to be qualified to the metrics where MixSA is actually best (LPIPS, PSNR, user-study preference) and the FID gap should be discussed. As written, the paper's own numbers contradict its main quantitative claim.
  4. [V.A, V.D, Tables I-III] The parameter-selection protocol is not described as a valid evaluation protocol. Section V.A states that parameters were determined using 'the same validation sets' as Semi-ref2Sketch and that the parameters were 'adjusted using a single test style across all experiments'. If ζ, β, α, attention-injection layers, and binarization thresholds are tuned using test styles, the reported LPIPS/FID/PSNR values are optimistic and the comparisons with baseline methods are not apples-to-apples. Please describe a held-out validation split or report a sensitivity analysis, and state exactly which parameter settings were used for each dataset and table.
  5. [IV.B, Eq. (5)] The entire query-mixing mechanism rests on the assumption that latent query features of a colored image and an edge map can be added and subtracted in the latent space. Because the downstream self-attention uses a Softmax, a convex combination of queries does not automatically produce a coherent combination of contour and texture; this is an empirical assumption. The only evidence provided in the paper is the final output images. Please add a direct test of the linearity assumption—for example, measure contour fidelity (e.g., overlap with the TEED edge map) as β varies, or compare Eq. (5) with a normalized alternative—so that readers can judge whether the assumption holds.
  6. [III.B, Eq. (2)] Eq. (2) as written is not the standard DDIM update. For deterministic DDIM, the update includes an additional predicted-noise term, sqrt(1−alpha_{t−1}) eps_theta, which is missing from Eq. (2); moreover, the text says the inversion 'estimates xt−1 from xt', whereas DDIM inversion moves from x0 toward xT. Since DDIM inversion is the foundation of the whole pipeline, the equation and the surrounding terminology should be corrected.
minor comments (6)
  1. [IV.C.2] The text says β is introduced in 'Equation 9', but the β mixture is defined in Eq. (5); the equation numbering in this section should be corrected.
  2. [IV.D, V.E] The abbreviation RCD (Reconstruction Color Distribution) is used in Table III and Fig. 13 but is only defined in Section V.E; Section IV.D should introduce and name this component when the binarization and contrast-adjustment procedure is described.
  3. [V.A] The default α=0.55 for 'strokes sparse thresholds' is mentioned but α is never defined in the method; a formal definition of α and its role in the pipeline is needed.
  4. [V.F, Table IV] The user study reports raw preference percentages without statistical significance testing or confidence intervals; a binomial test or a bootstrapped confidence interval would make the 71.68% result substantially more convincing.
  5. [Introduction and Abstract] The claim of creating 'novel, unseen styles' should be softened: Eq. (7) is by construction a convex combination of query features, so interpolation is a direct property of the formula. The contribution is better framed as providing a convenient control axis whose outputs still require perceptual validation.
  6. [General] The paper does not mention a code release. Since MixSA is training-free and built from public components, releasing code or detailed pseudocode for the attention injection and the parameter sweeps would substantially help reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

The core K/V-injection pipeline is empirically self-contained, but the advertised 'interpolation between styles to create novel, unseen styles' reduces by construction to the convex combination in Eqs. (6)-(7); Eq. (6)'s ζ semantics are also reversed in the text, a reproducibility defect rather than an additional circularity.

  1. self definitional [Abstract and Section IV.C ('Sketch Style Interpolation'), Eqs. (6)-(7)]
    "'...enabling interpolation between styles to create novel, unseen styles.' 'Qm_t = ζ × Qcs_t + (1−ζ) × Qr_t (6)' 'ϕs_out = Attn(ζ ×(β×Qc_t +(1−β)×Qs_t)+(1−ζ)×Qr_t, Kr_t, Vr_t) (7)'"

    The advertised interpolation capability is identical to the definition of the query feature Qm_t. Eq. (6) defines Qm_t as a convex combination of Qcs_t and Qr_t, and Eq. (7) is simply that definition substituted into the attention operator. Therefore any 'novel, unseen style' produced by varying ζ (or β) is, by construction, the attention output over the user-chosen weighted average of latent queries; it is not a derived, predicted, or independently fitted property. This makes one headline contribution self-definitional. The remainder of the method—K/V injection, DDIM inversion of edge/color/reference latents, binarization, and U2-Net foregrounding—is empirically evaluated and is not circular.

full rationale

MixSA is primarily a constructive, training-free image-editing pipeline rather than a first-principles derivation. The K/V substitution in Eq. (4), the DDIM-inverted edge initialization, and the post-processing are explicit mechanisms, and their quantitative/user-study evaluation is independent external support. There are no load-bearing self-citations: the cited style-transfer priors (StyleAligned, StyleID, Swapping Self-Attention) are prior external works, not the present authors' own unpublished claims. The paper also states its latent-linearity premise explicitly ('We assume that features of the colored image can be additively and subtractively manipulated in the latent space', Section IV.B); this is an unvalidated assumption inherited by Eq. (5), but it is an assumption, not a circular reduction. The one genuine circularity concern is the 'interpolation between styles / novel unseen styles' claim, which reduces by construction to the convex combination in Eqs. (6)-(7). I additionally flag that Section IV.C.1 and Fig. 5 describe higher ζ as increasing reference preservation, while Eq. (6) weights Qcs_t by ζ, so the textual control semantics are reversed relative to the equation; this is an internal inconsistency/reproducibility defect rather than a circularity and should be weighed as a correctness risk. Because the core sketch-extraction result is self-contained and benchmarked, while one headline sub-claim is definitional, the appropriate score is 4.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method rests on prior attention-swapping style transfer, DDIM inversion, and pretrained Stable Diffusion. Its own contributions are the query mixing parameters (zeta, beta), layer selection, edge detector choice, and post-processing steps, all of which are hand-tuned. No new physical or conceptual entities are introduced. The central claim depends on the unchecked linearity of the latent space and on the effectiveness of KV substitution, both inherited from the prior diffusion-based style transfer literature.

free parameters (6)
  • zeta (reference alignment weight) = 0.4 (default)
    Mixes query features between the color/edge composite and the reference; hand-tuned per dataset and test style.
  • beta (texture retention weight) = 0.5 (default)
    Mixes query features between the colored image and the edge map to control texture inclusion.
  • alpha (strokes sparse threshold) = 0.55
    Default threshold for edge sparsity; hand-set in the experimental settings.
  • attention injection layers = 10th and 11th decoder layers
    Chosen by experiment; the paper states 'we discovered that injecting features at the 10th and 11th layers yielded the best results'.
  • binarization threshold = 230
    Pixels with brightness above 230 are binarized to counter gray averaging in Stable Diffusion.
  • edge detection method = TEED (default)
    Choice among Canny, HED, and TEED affects the initial outlines; TEED is selected as the default.
assumptions (4)
  • domain assumption Features of colored and edge images in the latent space can be additively and subtractively manipulated
    Section IV.B states: 'We assume that features of the colored image can be additively and subtractively manipulated in the latent space.' This underlies Eq. (5) and the entire query mixing procedure.
  • domain assumption Substituting the reference sketch's keys and values in self-attention transfers brushstroke style
    Borrowed from StyleID, StyleAligned, and Swapping Self-Attention; the paper relies on this without providing a proof or independent validation.
  • ad hoc to paper High-frequency components accumulate more noise variance and are therefore harder to reconstruct
    Section IV.D uses this to justify the edge-map initialization, but the statement is qualitative and the variance argument in Eqs. (9)-(10) is not rigorously defined.
  • domain assumption Stable Diffusion v1.4 can decode the inverted edge-map latent into a meaningful sketch
    The method relies on the pretrained SD autoencoder and U-Net to generate sketches from the edge-map-initialized latent without any fine-tuning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MixSA: Training-free Reference-based Sketch Extraction via Mixture-of-Self-Attention." pith.science (2026). https://pith.science/paper/WK6HSY3M

@misc{pith2026250100816,
  author       = {Pith},
  title        = {Pith review of: MixSA: Training-free Reference-based Sketch Extraction via Mixture-of-Self-Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WK6HSY3M}},
  note         = {Machine review of arXiv:2501.00816}
}
read the original abstract

Current sketch extraction methods either require extensive training or fail to capture a wide range of artistic styles, limiting their practical applicability and versatility. We introduce Mixture-of-Self-Attention (MixSA), a training-free sketch extraction method that leverages strong diffusion priors for enhanced sketch perception. At its core, MixSA employs a mixture-of-self-attention technique, which manipulates self-attention layers by substituting the keys and values with those from reference sketches. This allows for the seamless integration of brushstroke elements into initial outline images, offering precise control over texture density and enabling interpolation between styles to create novel, unseen styles. By aligning brushstroke styles with the texture and contours of colored images, particularly in late decoder layers handling local textures, MixSA addresses the common issue of color averaging by adjusting initial outlines. Evaluated with various perceptual metrics, MixSA demonstrates superior performance in sketch quality, flexibility, and applicability. This approach not only overcomes the limitations of existing methods but also empowers users to generate diverse, high-fidelity sketches that more accurately reflect a wide range of artistic expressions.

Figures

Figures reproduced from arXiv: 2501.00816 by the authors.

Figure 1
Figure 1. We propose MixSA, a training-free approach for extracting sketches from a color image using an input reference style [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Data-driven sketch extraction methods (a) struggle [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The architecture of our proposed MixSA begins with a color image [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Detailed mechanism of the mixture-of-self-attention [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Illustration of ζ as the control axis for the degree of freehand (Xieyi) style. In the first row, with β = 0 (no texture), the scene’s sketch is automatically extracted based on object contours. From left to right, the sketch becomes increasingly freehand as ζ increase…
Figure 7
Figure 7. Figure 7: Visualization of the contributions of Qc t , Qs t , and Qr t to ϕ s out under varying ζ and β values. The plots illustrate how dif￾ferent parameter settings influence the final output, with color gradients representing the magnitude of each contribution. provides flexi…
Figure 8
Figure 8. Figure 8: Illustration of the challenges faced by existing dif [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 10
Figure 10. Figure 10: Various examples generated by our method compared to baseline methods. While Semi-ref2sketch produces high [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 12
Figure 12. Figure 12: Illustration of sketch extraction on the FS2K [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Illustration of sketch extraction in the ablation study. The “Initial + MSA + DCT + RCD” configuration achieves the [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 14
Figure 14. Figure 14: Art drawing generated by adjusting the abstraction [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]
Figure 15
Figure 15. Figure 15: Portrait drawing generated by adjusting the abstraction [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]
Figure 16
Figure 16. Figure 16: Examples of failure cases where light or low-contrast [PITH_FULL_IMAGE:figures/full_fig_p014_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 34 canonical work pages

  1. [1]

    The tao of painting: A study of the ritual disposition of chinese painting,

    J. F. Cahill, “The tao of painting: A study of the ritual disposition of chinese painting,” 1959

    work page 1959

  2. [2]

    Style and abstraction in portrait sketching,

    I. Berger, A. Shamir, M. Mahler, E. Carter, and J. Hodgins, “Style and abstraction in portrait sketching,” ACM Transactions on Graphics (TOG), vol. 32, no. 4, pp. 1–12, 2013

    work page 2013

  3. [3]

    A computational approach to edge detection,

    J. Canny, “A computational approach to edge detection,” IEEE Transac- tions on pattern analysis and machine intelligence , no. 6, pp. 679–698, 1986

    1986

  4. [4]

    Holistically-nested edge detection,

    S. Xie and Z. Tu, “Holistically-nested edge detection,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1395– 1403

    work page 2015

  5. [5]

    Tiny and efficient model for the edge detection generalization,

    X. Soria, Y . Li, M. Rouhani, and A. D. Sappa, “Tiny and efficient model for the edge detection generalization,” in CVPR, 2023, pp. 1364–1373

    work page 2023

  6. [6]

    L. Zhang. (2017) sketchkeras: an u-net with some algorithm to take sketch from paints. GitHub repository. [Online]. Available: https://github.com/lllyasviel/sketchKeras

    work page 2017

  7. [7]

    Adversarial open domain adaptation for sketch-to-photo synthesis,

    X. Xiang, D. Liu, X. Yang, Y . Zhu, X. Shen, and J. P. Allebach, “Adversarial open domain adaptation for sketch-to-photo synthesis,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 1434–1444

    work page 2022

  8. [8]

    Clipasso: Semantically-aware object sketching,

    Y . Vinker, E. Pajouheshgar, J. Y . Bo, R. C. Bachmann, A. H. Bermano, D. Cohen-Or, A. Zamir, and A. Shamir, “Clipasso: Semantically-aware object sketching,” ACM Transactions on Graphics (TOG), vol. 41, no. 4, pp. 1–11, 2022

    work page 2022

Show all 56 references
  1. [9]

    Clipascene: Scene sketching with different types and levels of abstraction,

    Y . Vinker, Y . Alaluf, D. Cohen-Or, and A. Shamir, “Clipascene: Scene sketching with different types and levels of abstraction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4146–4156

    2023

  2. [10]

    Reference based sketch extraction via attention mechanism,

    A. Ashtari, C. W. Seo, C. Kang, S. Cha, and J. Noh, “Reference based sketch extraction via attention mechanism,” ACM Trans. Graph., vol. 41, no. 6, nov 2022. [Online]. Available: https: //doi.org/10.1145/3550454.3555504

    2022

  3. [11]

    Semi-supervised reference-based sketch extraction using a contrastive learning framework,

    C. W. Seo, A. Ashtari, and J. Noh, “Semi-supervised reference-based sketch extraction using a contrastive learning framework,” ACM Trans- actions on Graphics (TOG) , vol. 42, no. 4, pp. 1–12, 2023

    2023

  4. [12]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models,

    H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang, “Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models,” arXiv preprint arXiv:2308.06721 , 2023

    2023 arXiv

  5. [13]

    Style aligned image generation via shared attention,

    A. Hertz, A. V oynov, S. Fruchter, and D. Cohen-Or, “Style aligned image generation via shared attention,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 4775–4785

    2024

  6. [14]

    Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer,

    J. Chung, S. Hyun, and J.-P. Heo, “Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 8795–8805

    2024

  7. [15]

    Instantstyle: Free lunch towards style-preserving in text-to-image generation,

    H. Wang, Q. Wang, X. Bai, Z. Qin, and A. Chen, “Instantstyle: Free lunch towards style-preserving in text-to-image generation,” arXiv preprint arXiv:2404.02733, 2024

    2024 arXiv

  8. [16]

    Bi-directional cascade network for perceptual edge detection,

    J. He, S. Zhang, M. Yang, Y . Shan, and T. Huang, “Bi-directional cascade network for perceptual edge detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 3828–3837

    2019

  9. [17]

    The trea- sure beneath multiple annotations: An uncertainty-aware edge detector,

    C. Zhou, Y . Huang, M. Pu, Q. Guan, L. Huang, and H. Ling, “The trea- sure beneath multiple annotations: An uncertainty-aware edge detector,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 507–15 517

    2023

  10. [18]

    Dense extreme inception network: Towards a robust cnn model for edge detection,

    X. S. Poma, E. Riba, and A. Sappa, “Dense extreme inception network: Towards a robust cnn model for edge detection,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision , 2020, pp. 1923–1932

    2020

  11. [19]

    APDrawingGAN: Generat- ing artistic portrait drawings from face photos with hierarchical gans,

    R. Yi, Y .-J. Liu, Y .-K. Lai, and P. L. Rosin, “APDrawingGAN: Generat- ing artistic portrait drawings from face photos with hierarchical gans,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR ’19), 2019, pp. 10 743–10 752

    2019

  12. [20]

    Neural probabilistic graphical model for face sketch synthesis,

    M. Zhang, N. Wang, Y . Li, and X. Gao, “Neural probabilistic graphical model for face sketch synthesis,” IEEE Transactions on Neural Networks and Learning Systems , vol. 31, no. 7, pp. 2623–2637, 2020

    2020

  13. [21]

    Dual-transfer face sketch–photo synthesis,

    M. Zhang, R. Wang, X. Gao, J. Li, and D. Tao, “Dual-transfer face sketch–photo synthesis,” IEEE Transactions on Image Processing , vol. 28, no. 2, pp. 642–657, 2019. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 15

    2019

  14. [22]

    Deep latent low-rank represen- tation for face sketch synthesis,

    M. Zhang, N. Wang, Y . Li, and X. Gao, “Deep latent low-rank represen- tation for face sketch synthesis,” IEEE Transactions on Neural Networks and Learning Systems , vol. 30, no. 10, pp. 3109–3123, 2019

    2019

  15. [23]

    Bionic face sketch generator,

    ——, “Bionic face sketch generator,” IEEE Transactions on Cybernetics, vol. 50, no. 6, pp. 2701–2714, 2020

    2020

  16. [24]

    A unified arbitrary style transfer framework via adaptive contrastive learning,

    Y . Zhang, F. Tang, W. Dong, H. Huang, C. Ma, T.-Y . Lee, and C. Xu, “A unified arbitrary style transfer framework via adaptive contrastive learning,” ACM Transactions on Graphics , 2023

    2023

  17. [25]

    Image style transfer using convolutional neural networks,

    L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 2414–2423

    2016

  18. [26]

    Arbitrary style transfer in real-time with adaptive instance normalization,

    X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in Proceedings of the IEEE interna- tional conference on computer vision , 2017, pp. 1501–1510

    2017

  19. [27]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695

    2022

  20. [28]

    Painterly style transfer with learned brush strokes,

    X.-C. Liu, Y .-C. Wu, and P. Hall, “Painterly style transfer with learned brush strokes,” IEEE Transactions on Visualization and Computer Graphics, 2023

    2023

  21. [29]

    Inversion-based style transfer with diffusion models,

    Y . Zhang, N. Huang, F. Tang, H. Huang, C. Ma, W. Dong, and C. Xu, “Inversion-based style transfer with diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 10 146–10 156

    2023

  22. [30]

    Zero-shot contrastive loss for text- guided diffusion image style transfer,

    S. Yang, H. Hwang, and J. C. Ye, “Zero-shot contrastive loss for text- guided diffusion image style transfer,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22 873–22 882

    2023

  23. [31]

    Rast: Restorable arbitrary style transfer via multi-restoration,

    Y . Ma, C. Zhao, X. Li, and A. Basu, “Rast: Restorable arbitrary style transfer via multi-restoration,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2023, pp. 331–340

    2023

  24. [32]

    Null- text inversion for editing real images using guided diffusion models,

    R. Mokady, A. Hertz, K. Aberman, Y . Pritch, and D. Cohen-Or, “Null- text inversion for editing real images using guided diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 6038–6047

    2023

  25. [33]

    Fatezero: Fusing attentions for zero-shot text-based video editing,

    C. Qi, X. Cun, Y . Zhang, C. Lei, X. Wang, Y . Shan, and Q. Chen, “Fatezero: Fusing attentions for zero-shot text-based video editing,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 932–15 942

    2023

  26. [34]

    Stylediffusion: Controllable dis- entangled style transfer via diffusion models,

    Z. Wang, L. Zhao, and W. Xing, “Stylediffusion: Controllable dis- entangled style transfer via diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 7677–7689

    2023

  27. [35]

    Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation,

    Y . Wei, Y . Zhang, Z. Ji, J. Bai, L. Zhang, and W. Zuo, “Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 15 943–15 953

    2023

  28. [36]

    Cross-image attention for zero-shot appearance transfer,

    Y . Alaluf, D. Garibi, O. Patashnik, H. Averbuch-Elor, and D. Cohen-Or, “Cross-image attention for zero-shot appearance transfer,”arXiv preprint arXiv:2311.03335, 2023

    2023 arXiv

  29. [37]

    Visual style prompting with swapping self-attention,

    J. Jeong, J. Kim, Y . Choi, G. Lee, and Y . Uh, “Visual style prompting with swapping self-attention,” arXiv preprint arXiv:2402.12974 , 2024

    2024 arXiv

  30. [38]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international con- ference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 ...

    2015

  31. [39]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  32. [40]

    Freeu: Free lunch in diffusion u-net,

    C. Si, Z. Huang, Y . Jiang, and Z. Liu, “Freeu: Free lunch in diffusion u-net,” in CVPR, 2024

    2024

  33. [41]

    Diffusion with offset noise,

    C. Labs, “Diffusion with offset noise,” 2023, accessed: 2024-05-17. [Online]. Available: https://www.crosslabs.org/blog/ diffusion-with-offset-noise

    2023

  34. [42]

    U2-net: Going deeper with nested u-structure for salient object detection,

    X. Qin, Z. Zhang, C. Huang, M. Dehghan, O. R. Zaiane, and M. Jager- sand, “U2-net: Going deeper with nested u-structure for salient object detection,” Pattern recognition, vol. 106, p. 107404, 2020

    2020

  35. [43]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,

    L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE transactions on pattern analysis and machine intelligence , vol. 40, no. 4, pp. 834–848, 2017

    2017

  36. [44]

    Polarmask: Single shot instance segmentation with polar rep- resentation,

    E. Xie, P. Sun, X. Song, W. Wang, X. Liu, D. Liang, C. Shen, and P. Luo, “Polarmask: Single shot instance segmentation with polar rep- resentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 12 193–12 202

    2020

  37. [45]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502 , 2020

    2010 arXiv

  38. [46]

    Facial- sketch synthesis: a new challenge,

    D.-P. Fan, Z. Huang, P. Zheng, H. Liu, X. Qin, and L. Van Gool, “Facial- sketch synthesis: a new challenge,” Machine Intelligence Research , vol. 19, no. 4, pp. 257–287, 2022

    2022

  39. [47]

    Anime sketch colorization pair,

    T. B. Kang, “Anime sketch colorization pair,” 2018, accessed: 2024- 05-18. [Online]. Available: https://www.kaggle.com/datasets/ktaebum/ anime-sketch-colorization-pair/data

    2018

  40. [48]

    Image quality assessment: from error visibility to structural similarity,

    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing , vol. 13, no. 4, pp. 600–612, 2004

    2004

  41. [49]

    The unreasonable effectiveness of deep features as a perceptual metric,

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 586–595

    2018

  42. [50]

    Contour detection based on nonclassical receptive field inhibition,

    C. Grigorescu, N. Petkov, and M. A. Westenberg, “Contour detection based on nonclassical receptive field inhibition,” IEEE Transactions on image processing, vol. 12, no. 7, pp. 729–739, 2003

    2003

  43. [51]

    De- mystifying MMD GANs,

    M. Bi ´nkowski, D. J. Sutherland, M. Arbel, and A. Gretton, “De- mystifying MMD GANs,” in International Conference on Learning Representations, 2018

    2018

  44. [52]

    Pros and cons of gan evaluation measures,

    A. Borji, “Pros and cons of gan evaluation measures,” Computer vision and image understanding , vol. 179, pp. 41–65, 2019

    2019

  45. [53]

    Unaligned image-to-image translation by learning to reweight,

    S. Xie, M. Gong, Y . Xu, and K. Zhang, “Unaligned image-to-image translation by learning to reweight,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14 174–14 184

    2021

  46. [54]

    Swapping autoencoder for deep image manipulation,

    T. Park, J.-Y . Zhu, O. Wang, J. Lu, E. Shechtman, A. Efros, and R. Zhang, “Swapping autoencoder for deep image manipulation,” Ad- vances in Neural Information Processing Systems , vol. 33, pp. 7198– 7211, 2020

    2020

  47. [55]

    Multimodal unsu- pervised image-to-image translation,

    X. Huang, M.-Y . Liu, S. Belongie, and J. Kautz, “Multimodal unsu- pervised image-to-image translation,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 172–189

    2018

  48. [56]

    Learning to generate line drawings that convey geometry and semantics,

    C. Chan, F. Durand, and P. Isola, “Learning to generate line drawings that convey geometry and semantics,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 7915–7925. Rui Yang (Student Member, IEEE) received an M.S.Eng. degree in...

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.