{"id":"4b38990a-ad22-4c8c-8487-7139ee32d121","arxiv_id":"2501.00816","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A training-free method transfers a reference sketch's stroke style onto a photo's edge map by replacing self-attention keys and values in Stable Diffusion, with two sliders for texture and style.","lead":"MixSA extracts sketches from photos using a reference sketch's brushstroke style without any training, by swapping attention features inside a pretrained Stable Diffusion model. It lets users blend two styles and dial texture and abstraction up or down, which could give artists fast style control.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (6) and the surrounding text specify opposite behavior for the ζ control axis, so the paper's central 'precise control and interpolation' claim is not currently well defined.","rationale":"The paper's central claim is that MixSA is a training-free reference-based sketch extractor with precise, interpolable style control. For that claim to hold, the two control parameters ζ and β must behave as described: ζ should monotonically increase reference-sketch influence and β should monotonically increase texture retention. The manuscript contradicts the first requirement: Eq. 6 sets Qm = ζ Qcs + (1−ζ) Qr, so larger ζ means less reference content, while Sections IV.C.1 and IV.C.3 and Fig. 5 state the opposite. One of these must be wrong; until resolved, the user cannot know what the qualitative interpolation figures mean, and the 'precise control' contribution is not well defined. The reader's weakest_assumption about additive latent features is related: Eq. 5's linear blending of Qc and Qs is the other half of the control mechanism, and it is not verified beyond final images. I therefore treat the two as one concern: the query-mixing control mechanism is under-specified and contradicted by the text. The proposed grid experiment would settle the direction of both controls. I do not see a fatal flaw in the core K/V-swapping idea; the contradictions and missing validation are fixable, so the existing conditional verdict stands.","tokens_in":19554,"tokens_out":9701,"duration_ms":95271,"concrete_test":"Run the described MixSA pipeline (or reimplement Eqs. 4-6 on SD1.4 with DDIM inversion) on one fixed color image, one fixed TEED outline, and one reference sketch, keeping β = 0.5 fixed. Vary ζ over {0.0, 0.25, 0.5, 0.75, 1.0} and measure LPIPS distance from each output to (i) the reference sketch and (ii) the initial edge outline. If LPIPS-to-reference decreases monotonically as ζ increases, Eq. 6 is correct and the text/captions are wrong; if it increases, Eq. 6 must be inverted. Repeat with β over {0.0, 0.5, 1.0} at fixed ζ to confirm whether increasing β actually increases texture retention relative to the outline, as claimed in Section IV.B.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.C.1 defines Qm_t = ζ × Qcs_t + (1−ζ) × Qr_t (Eq. 6), which means ζ=1 produces the pure colored/edge query and ζ=0 produces the pure reference query. The text immediately says the opposite: 'A higher ζ value results in greater preservation of Qr_t', and Section IV.C.3 repeats that higher ζ 'enhances the preservation of the reference’s brushstroke styles'. The caption of Fig. 5 also states that increasing ζ makes strokes approach the reference sketch. This is not a minor typo: ζ is the paper's claimed axis for reference adherence and style interpolation, and the same reversed semantics would affect Fig. 7, the parameter guidance, and the interpolation examples. If the implemented code follows Eq. 6, the textual guidance and figure captions are reversed; if it follows the text, Eq. 6 is wrong and the interpolation results need re-interpretation. The nearest underlying mechanism, the DCT module in Section IV.B, further assumes that query features of the color image and edge map can be linearly blended (Eq. 5), but self-attention involves a nonlinear Softmax, so this linearity is not automatic and is not directly validated. Together these issues mean the central claim of 'precise control over texture density and enabling interpolation between styles' is not reproducible from the written method as it stands.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces MixSA, a training-free method for extracting a sketch from a color photograph in the style of a user-supplied reference sketch. The pipeline uses DDIM inversion to obtain latents for the color image, a TEED edge map, and the reference sketch; during denoising it replaces the self-attention keys and values of the edge-map stream with those of the reference, and forms the queries as a weighted combination of color, edge, and reference queries (Eqs. 5-7). The paper claims precise control over texture density and reference adherence via parameters ζ and β, interpolation between styles to create unseen styles, and a solution to the 'color averaging' artifact of Stable Diffusion. Quantitative comparisons are reported on 4SKST, FS2K, and Anime, together with an ablation study and a user study.","tokens_in":19839,"tokens_out":11879,"duration_ms":110167,"significance":"MixSA addresses a real limitation: prior sketch-extraction models are tied to training styles, while prior training-free style transfer does not preserve line and contour structure. The attention-swapping mechanism is simple, plausible, and supported by the qualitative figures: the reference brushstroke structure appears in the outputs, and the ζ/β axes visibly change texture density and reference adherence. The user study (71.68%) is suggestive, and the FS2K ablation shows that each component contributes. These strengths make the method a reasonable candidate for TVCG. However, as written the paper contains several load-bearing problems: a direct contradiction between Eq. (6) and the text/figures for the central control parameter ζ; an incorrect frequency-domain justification for the color-averaging analysis; a headline 'superior performance' claim contradicted by the FID result in Table I; a parameter-tuning protocol that does not establish generalization; and an unvalidated linearity assumption at the core of the query-mixing formula. These issues prevent acceptance in the current form.","major_comments":[{"comment":"The semantics of ζ are stated in two opposite directions. Eq. (6) defines Qm_t = ζ Qcs_t + (1−ζ) Qr_t, so ζ=1 gives the pure contour/color query and ζ=0 gives the pure reference query. The text immediately after Eq. (6) says 'A higher ζ value results in greater preservation of Qr_t', Section IV.C.3 repeats that higher ζ enhances preservation of the reference brushstroke styles, and the caption of Fig. 5 says increasing ζ makes strokes approach the reference sketch. These statements would be correct for (1−ζ), not ζ, and the same reversal appears in the supplementary Fig. A2 caption. Because ζ is presented as the paper's precise control and interpolation axis, the method is not reproducible as written. Please either change Eq. (6) to Qm_t = (1−ζ) Qcs_t + ζ Qr_t or revise all textual descriptions and figure captions, and then re-examine Fig. 7 and the supplementary parameter plots accordingly.","section":"IV.C.1 (Eq. 6), IV.C.3, Fig. 5"},{"comment":"The frequency-domain argument is not correct. In Eq. (8), the noise ε is i.i.d. Gaussian; under any orthonormal frequency decomposition the noise variance is the same for each frequency component, so the cumulative variances in Eq. (9) are equal, not related by ≫. The correct intuition is that natural images have most of their signal power at low frequencies, so high-frequency components have a lower signal-to-noise ratio and are harder to reconstruct; it is not that high-frequency components accumulate more noise variance. Since this subsection is used to justify the initial-contour anchoring and binarization (the third claimed contribution), the argument should be corrected or reframed.","section":"IV.D, Eqs. (8)-(10)"},{"comment":"The abstract and introduction claim 'superior performance' and results that surpass other methods 'in both quality and fidelity', but in Table I MixSA's FID on 4SKST is 89.04, worse than Semi-ref2sketch's 82.18. FID is one of the paper's headline metrics, so the quantitative superiority claim needs to be qualified to the metrics where MixSA is actually best (LPIPS, PSNR, user-study preference) and the FID gap should be discussed. As written, the paper's own numbers contradict its main quantitative claim.","section":"V.D, Table I; Abstract"},{"comment":"The parameter-selection protocol is not described as a valid evaluation protocol. Section V.A states that parameters were determined using 'the same validation sets' as Semi-ref2Sketch and that the parameters were 'adjusted using a single test style across all experiments'. If ζ, β, α, attention-injection layers, and binarization thresholds are tuned using test styles, the reported LPIPS/FID/PSNR values are optimistic and the comparisons with baseline methods are not apples-to-apples. Please describe a held-out validation split or report a sensitivity analysis, and state exactly which parameter settings were used for each dataset and table.","section":"V.A, V.D, Tables I-III"},{"comment":"The entire query-mixing mechanism rests on the assumption that latent query features of a colored image and an edge map can be added and subtracted in the latent space. Because the downstream self-attention uses a Softmax, a convex combination of queries does not automatically produce a coherent combination of contour and texture; this is an empirical assumption. The only evidence provided in the paper is the final output images. Please add a direct test of the linearity assumption—for example, measure contour fidelity (e.g., overlap with the TEED edge map) as β varies, or compare Eq. (5) with a normalized alternative—so that readers can judge whether the assumption holds.","section":"IV.B, Eq. (5)"},{"comment":"Eq. (2) as written is not the standard DDIM update. For deterministic DDIM, the update includes an additional predicted-noise term, sqrt(1−alpha_{t−1}) eps_theta, which is missing from Eq. (2); moreover, the text says the inversion 'estimates xt−1 from xt', whereas DDIM inversion moves from x0 toward xT. Since DDIM inversion is the foundation of the whole pipeline, the equation and the surrounding terminology should be corrected.","section":"III.B, Eq. (2)"}],"minor_comments":[{"comment":"The text says β is introduced in 'Equation 9', but the β mixture is defined in Eq. (5); the equation numbering in this section should be corrected.","section":"IV.C.2"},{"comment":"The abbreviation RCD (Reconstruction Color Distribution) is used in Table III and Fig. 13 but is only defined in Section V.E; Section IV.D should introduce and name this component when the binarization and contrast-adjustment procedure is described.","section":"IV.D, V.E"},{"comment":"The default α=0.55 for 'strokes sparse thresholds' is mentioned but α is never defined in the method; a formal definition of α and its role in the pipeline is needed.","section":"V.A"},{"comment":"The user study reports raw preference percentages without statistical significance testing or confidence intervals; a binomial test or a bootstrapped confidence interval would make the 71.68% result substantially more convincing.","section":"V.F, Table IV"},{"comment":"The claim of creating 'novel, unseen styles' should be softened: Eq. (7) is by construction a convex combination of query features, so interpolation is a direct property of the formula. The contribution is better framed as providing a convenient control axis whose outputs still require perceptual validation.","section":"Introduction and Abstract"},{"comment":"The paper does not mention a code release. Since MixSA is training-free and built from public components, releasing code or detailed pseudocode for the attention injection and the parameter sweeps would substantially help reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central mechanism is promising and, in my view, the paper is salvageable: the attention-swapping idea is simple, the qualitative results are attractive, and the user study is encouraging. The main problems are internal inconsistencies (the ζ reversal in Eq. (6) versus the text and figures), an overstated quantitative claim contradicted by Table I, an unsupported frequency-domain argument, and an inadequately described parameter-tuning protocol. These are fixable in a careful revision, so I do not see grounds for rejection. I would also encourage the editor to ask for code or pseudocode, because the method's reproducibility would otherwise rely on the reader correctly guessing how the many hyperparameters were chosen."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth reading and worth sending to referees, but the authors need to fix a real internal contradiction before it's publishable. The core idea is a sensible extension of attention swapping: take a reference sketch's self-attention keys/values, inject them into a Stable Diffusion denoising process, initialize from an edge map (TEED) instead of random noise, and mix query features from the color image and edge map with two weights (β and ζ). I don't think the individual pieces are new—StyleID, StyleAligned, and Swapping Self-Attention all do KV substitution—but the specific combination for reference-based sketch extraction is new, and the qualitative results genuinely show the claimed behavior. The brushstroke transfer looks convincing, and the ability to interpolate between two styles is a nice practical feature. The paper also cites the relevant prior work fairly.\n\nThe biggest problem is the ζ parameter. Equation (6) defines Qm_t = ζ * Qcs_t + (1−ζ) * Qr_t, so larger ζ means more of the colored/edge query and less of the reference. But Section IV.C.1 says a higher ζ results in greater preservation of Qr_t, and Section IV.C.3 repeats that higher ζ enhances reference brushstroke preservation. Fig. 5's caption says the same. This is not a typo: ζ is the paper's central control axis for style interpolation, so the current text makes the method non-reproducible as written. The authors need to decide which direction they actually implemented and rewrite the equations or the text consistently.\n\nSecond, the paper leans on an unvalidated linearity assumption: that colored-image and edge-map features can be added and subtracted in latent space. They state it explicitly in Section IV.B, but don't test it directly. Given self-attention uses softmax, this is not automatic. The qualitative results suggest it works in practice, but \"works in practice\" is not the same as a validated assumption.\n\nThird, the quantitative evidence is mixed. On 4SKST, FID is 89.04 vs Semi-ref's 82.18, yet the abstract says the method demonstrates superior performance. That's accurate only if you cherry-pick LPIPS and PSNR. The color-averaging analysis in Section IV.D is hand-wavy—the frequency-domain argument doesn't directly support the binarization post-process. And there are many hand-tuned parameters (α, ζ, β, layer choices, binarization threshold) with no code release, which makes reproduction hard.\n\nWho's it for: anyone working on sketch extraction, diffusion-based image editing, or attention manipulation. It deserves a serious referee: the core idea is plausible, the qualitative evidence is strong, and the flaws are fixable. I'd recommend major revision, not rejection, and I'd specifically ask for code and a corrected parameter description.","headline":"A useful training-free sketch extraction method with convincing qualitative results, but the ζ control axis is described backwards in the text, and the quantitative and theoretical support is thinner than claimed.","tokens_in":20370,"tokens_out":2523,"would_cite":true,"duration_ms":22401,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By swapping self-attention keys and values inside a pretrained diffusion U-Net, this paper claims, sketches can be extracted from photos in any reference brushstroke style without any training.","keywords":["sketch extraction","reference-based style","self-attention manipulation","diffusion models","DDIM inversion","training-free image-to-image translation","stable diffusion","texture density control"],"falsifier":"Encode a color photo and its edge map separately with DDIM inversion, form the latent mixture $\\beta z_{\\text{color}} + (1-\\beta)z_{\\text{edge}}$ for several $\\beta$ values, decode each mixture back to an image, and compare against the same convex combination done directly in pixel space; if the decoded mixtures do not track the pixel-space blends monotonically, the additive latent-space assumption underlying Eq. (5) is violated.","tokens_in":19325,"feed_emoji":"✏️","tokens_out":7786,"duration_ms":65953,"temperature":0.7,"pith_summary":"This paper claims that sketch extraction from a color photo can be done with no training while faithfully copying the brushstroke style of an arbitrary reference sketch. The proposed method, MixSA, starts from the photo's edge map as the initial latent, then during denoising replaces the self-attention keys and values in late decoder layers of Stable Diffusion with those of the reference sketch, and mixes the query features from photo, edge map, and reference with two user-controlled weights. A binarization post-step counters Stable Diffusion's tendency to average colors toward mid-gray, preserving crisp black-and-white lines. If the claim holds, artists could generate sketches in any style, or blended styles, from a single reference image and a photo, without paired training data.","feed_headline":"Training-free sketch extraction copies any reference style","feed_subtitle":"MixSA swaps self-attention keys and values in Stable Diffusion to transfer brush strokes without retraining.","key_machinery":"The key mechanism is the mixture-of-self-attention operation, written as $\\phi^s_{\\text{out}} = \\text{Attn}(Q^m_t, K^r_t, V^r_t)$, where $K^r_t$ and $V^r_t$ are the reference sketch's self-attention keys and values taken from its DDIM-inverted denoising trajectory, and $Q^m_t = \\zeta(\\beta Q^c_t + (1-\\beta)Q^s_t) + (1-\\zeta)Q^r_t$ is a query fused from the colored image's query, the edge map's query, and the reference's query. This operation is the single point where brushstroke texture enters the generation; the Decomposing Contours and Texture (DCT) module supplies the $\\beta$-weighted query mixture, the initial-outline starting point supplies structure, and the binarization step supplies color correction. It is applied only in late decoder self-attention layers, which the paper identifies as the layers responsible for local textures.","core_discovery":"The central discovery the author is trying to establish is that reference brushstroke style can be transferred into a sketch purely by editing self-attention inside the denoising U-Net of a pretrained latent diffusion model. Concretely, the reference sketch's keys and values, captured via DDIM inversion, replace those of the sketch being generated, while the query is a weighted sum of the color photo's query, the edge map's query, and the reference's query, with weights $\\beta$ and $\\zeta$. The paper argues that late decoder self-attention layers around layers 10 and 11 handle local stroke textures, so injecting the reference's keys and values there is what aligns brushstroke style with the photo's contours. It further claims that using the photo's edge map as the starting latent, together with contrast-enhancing binarization of extreme brightness pixels, fixes the color-averaging artifact that otherwise washes out black-and-white sketches.","pith_inferences":["The same key/value substitution likely transfers other line-based visual attributes, such as hatching density or pen pressure variation, that the paper's style set does not explicitly test; this is my inference, not a claim of the paper.","The latent linearity assumption could be probed directly on the autoencoder latents, and if it holds only approximately, the $\\zeta$ and $\\beta$ dials might be re-parameterized as image-domain attention-mask interpolation.","Since the mechanism is backbone-agnostic, the same recipe should transfer to newer latent diffusion models by identifying the analogous late-decoder self-attention layers, though the paper only tests Stable Diffusion 1.4.","In practice, setting $\\zeta$ and $\\beta$ per style may require a small manual search; a learned or automatic parameter selector would be a natural follow-up, and is not addressed in the paper."],"forward_implications":["Users can extract a sketch in any reference style, including styles absent from any training set, by supplying one photo and one reference sketch at inference time.","The user dials $\\zeta$ and $\\beta$ give explicit control over reference adherence and texture retention, so the same pipeline spans sparse, abstract strokes to dense, detailed ones.","Blending two reference sketches by adjusting the query mixture yields intermediate, previously unseen sketch styles, as demonstrated with Xieyi and Gongbi styles.","Because no training is needed, switching to a different reference style is instantaneous at inference, unlike the paired-data baselines Ref2sketch and Semi-ref2sketch.","With foreground extraction enabled, the method can sketch only the salient subject, ignoring a distracting background."],"supporting_citations":[{"why":"Introduces swapping self-attention keys and values for style transfer, the direct inspiration for MixSA's KV substitution.","marker":"[37]"},{"why":"StyleAligned's shared attention shows that self-attention features can transfer style, grounding the claim that reference KV injection aligns textures.","marker":"[13]"},{"why":"StyleID is the training-free style-transfer baseline that MixSA extends, and a comparison method in the experiments.","marker":"[14]"},{"why":"Ref2sketch is the training-based reference sketch extraction method that MixSA aims to beat, and a baseline in tables and figures.","marker":"[10]"},{"why":"Semi-ref2sketch supplies the 4SKST dataset, the semi-supervised baseline, and the validation-set parameter-setting approach adopted for zeta and beta.","marker":"[11]"},{"why":"TEED is the default edge-detection algorithm that produces the initial contour map used as the structural anchor.","marker":"[5]"},{"why":"DDIM inversion and sampling convert the photo, edge map, and reference sketch into the latents whose self-attention features are manipulated.","marker":"[45]"},{"why":"The offset-noise fine-tuned checkpoint on black-and-white images is used in the color-distribution correction step to preserve sharpness.","marker":"[41]"},{"why":"U2-Net performs foreground extraction in the scene-decomposition step so sketches can focus on the salient subject.","marker":"[42]"}],"fun_headline_variants":["Training-free sketch style transfer via self-attention swap","MixSA: Copy sketch styles without any training","Swap self-attention keys to match reference sketch styles","Zero-training sketch extraction using diffusion priors","Reference style sketches by editing self-attention layers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that in the latent space of Stable Diffusion, the features of a colored image and an edge map can be linearly added and subtracted, so the $\\beta$-weighted query mixture behaves like a coherent blend of photo texture and edge contours; the paper states this assumption explicitly in Section IV.B.","fun_headline_variants_meta":{"raw":{"variants":["Training-free sketch style transfer via self-attention swap","MixSA: Copy sketch styles without any training","Swap self-attention keys to match reference sketch styles","Zero-training sketch extraction using diffusion priors","Reference style sketches by editing self-attention layers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1312,"prompt_tokens":917,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":533,"tokens_out":395,"duration_ms":3758,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:41:25.734362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Encode a color photo and its edge map separately with DDIM inversion, form the latent mixture $\\beta z_{\\text{color}} + (1-\\beta)z_{\\text{edge}}$ for several $\\beta$ values, decode each mixture back to an image, and compare against the same convex combination done directly in pixel space; if the decoded mixtures do not track the pixel-space blends monotonically, the additive latent-space assumption underlying Eq. (5) is violated.","supporting_citations":[{"cited_title":"Semi-supervised reference-based sketch extraction using a contrastive learning framework,","cited_arxiv_id":null,"evidence_quote":"Semi-ref2sketch supplies the 4SKST dataset, the semi-supervised baseline, and the validation-set parameter-setting approach adopted for zeta and beta."},{"cited_title":"Tiny and efficient model for the edge detection generalization,","cited_arxiv_id":null,"evidence_quote":"TEED is the default edge-detection algorithm that produces the initial contour map used as the structural anchor."},{"cited_title":"Diffusion with offset noise,","cited_arxiv_id":null,"evidence_quote":"The offset-noise fine-tuned checkpoint on black-and-white images is used in the color-distribution correction step to preserve sharpness."}],"review_version":1}