{"id":"05ed8e40-7271-4e6b-a059-33f30e61a759","arxiv_id":"2507.16732","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A training-free diffusion inpainting method that reweights self-attention for structural fidelity and propagates key-value style statistics from unmasked to masked regions, outperforming baselines on stylized benchmarks.","lead":"HarmonPaint is a new image inpainting method that works by changing how a diffusion model's attention layers handle masked regions, without any extra training. It aims to make newly generated content match the structure and artistic style of the surrounding image, which existing inpainting tools often fail to do.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SAMS replaces softmax attention with an unnormalized map; masked-token rows no longer sum to 1, so the claimed obj-bg masking also scales feature magnitudes, making the structural-fidelity mechanism ambiguous.","rationale":"The empirical claim that HarmonPaint beats trained baselines rests on two interventions. MAKVS's style assumption is empirically supported by ablations and is a design choice. The SAMS implementation, however, conflicts with the standard definition of attention: post-softmax masking without renormalization changes feature scale. This is an internal inconsistency, not merely a missing theoretical justification, and it is directly checkable. If the effect is due to scaling, the paper's core novelty (masking obj-bg interactions for structural fidelity) is not what is being measured. The reader's weakest_assumption focused on MAKVS; I partially disagree because the more concrete, load-bearing flaw is in SAMS. The conditional verdict remains appropriate, but the authors should add the normalization test and either renormalize or justify the scaling.","tokens_in":16106,"tokens_out":8315,"duration_ms":100176,"concrete_test":"Re-run the Stylized-COCO segmentation-mask experiment with SAMS modified to row-normalize Â_self after Eq. (6) (divide each row by its sum, using a small epsilon for empty rows), keeping all other hyperparameters (τ=0.1, λ=1.4, η=0.6, layers 2-6) identical. If the reported CS/IR/AS gains over SDI persist within approximately 0.1 CLIP Score, the normalization issue is not the operative mechanism. If the gains shrink or disappear, the unnormalized scaling is doing the work, and the paper's explanation of SAMS needs correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section IV-A, Eq. (4)-(6) construct Â_self = M_f M_f^T ⊙ A_self + (1-M_f)(1-M_f)^T ⊙ A_self, where A_self is the post-softmax attention matrix from Eq. (3). Because masking is applied after softmax, the rows of Â_self are not renormalized. For a masked token i (M_f(i)=1), the row of Â_self equals the original attention weights restricted to masked keys, so its row sum is the original attention mass assigned to the masked region, which is less than 1 whenever obj-bg attention is nonzero. For the soft-mask variant (Eq. 7), row sums are similarly uncontrolled. Thus the operation does not merely 'selectively mask obj-bg interactions'; it also scales down the features output for the masked region by the row sum before the residual connection. The paper's PCA evidence and the ablation in Table III are therefore consistent with an alternative explanation: the structural-fidelity gain comes from attenuating masked-region feature magnitudes, not from the semantic separation of object/background attention. Since the central claim is that a training-free attention-masking strategy achieves structural fidelity, the method as specified is ambiguous about which mechanism is responsible and is not fully reproducible without specifying whether row normalization is applied.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"HarmonPaint proposes a training-free extension of Stable Diffusion Inpainting that intervenes in self-attention to improve structural fidelity and stylistic harmony. The method has three main ingredients: a Self-Attention Masking Strategy (SAMS) that suppresses object-background attention in the U-Net encoder, a Mask-Adjusted Key-Value Strategy (MAKVS) that propagates mean-pooled key/value features from unmasked to masked regions, and an Attention Steer Loss that concentrates cross-attention on the masked region. The denoising schedule is also split into a structure-focused stage and a style-focused stage. Experiments are reported on two stylized datasets derived from MSCOCO and OpenImages, with quantitative metrics, ablations, a user study, and comparisons against BrushNet, PowerPaint, ControlNet Inpainting, Blended Latent Diffusion, and FLUX.","tokens_in":16363,"tokens_out":4419,"duration_ms":49521,"significance":"If the claims are substantiated, the paper would show that a fully training-free attention intervention can match or beat dedicated fine-tuned inpainting heads on stylized benchmarks, which is a practically valuable result. The paper has several strengths: the two mechanisms are cleanly separated, the qualitative comparisons are extensive, the authors include a user study, a comparison with FLUX, mask-sensitivity experiments, and an honest limitation statement for large masked regions. However, the current evidence is not yet conclusive because the evaluation benchmark is generated by the same StyleID mechanism that motivates MAKVS, and because the core equations contain dimensional ambiguities that make the method not fully reproducible as written.","major_comments":[{"comment":"The central MAKVS operation is dimensionally inconsistent as written. With Q, K, V of shape [HW, d], the term [K, λeK]^T has shape [2d, HW], so Q × [K, λeK]^T is not a valid matrix product unless Q is also expanded to [HW, 2d], which is not stated. Likewise, Eq. (12) writes the output as eA_self × [V; eV], which changes the output feature dimension to 2d without specifying any subsequent projection or reshaping. Since MAKVS is a load-bearing component, the authors should give exact tensor shapes (including heads and batch dimensions) or provide pseudocode. Without this, the method is not reproducible and the reported 'only key's mean' baseline in Fig. 3 cannot be checked.","section":"Sec. IV-B, Eqs. (11)-(12)"},{"comment":"Because masking is applied after softmax without row renormalization, SAMS does more than 'selectively mask obj-bg interactions': it also scales down the output features for masked-region tokens by the remaining row sum. The same issue applies to the soft-mask variant in Eq. (7), whose row sums are also uncontrolled. The PCA visualizations and Table III are therefore consistent with an alternative explanation in which the structural-fidelity gain comes from attenuating masked-region feature magnitudes rather than from separating object/background principal components. Please compare SAMS against a row-normalized version of Eqs. (4)-(6), and report either attention row sums or feature norms across layers and timesteps. If the row-normalized version retains the gains, the semantic-separation interpretation is supported; if not, the paper should be reworded.","section":"Sec. IV-A, Eqs. (4)-(7)"},{"comment":"The evaluation is circular in a way that favors the method. The Stylized-COCO and Stylized-OpenImages benchmarks are produced by applying StyleID to natural images, and MAKVS is directly motivated by StyleID's observation that K and V carry style. The method is therefore evaluated in-distribution for its core style-transfer assumption. Please add at least one benchmark that does not rely on StyleID, for example real stylized images (e.g., artwork with object masks), or synthetic stylization performed with a different model family (e.g., a GAN-based style transfer method), and report the same metrics per method. This is necessary to support the claim that HarmonPaint generalizes across diverse styles rather than only reproducing the style statistics of the model that generated the benchmark.","section":"Sec. V-A3 and Tables I-II"},{"comment":"The quantitative claims are not accompanied by any measure of variance or statistical testing. The datasets contain only 50 images per source, and several reported differences are small (e.g., CS 28.86 vs. 28.12 for PowerPaint in Table I; CMMD 0.103 vs. 0.109 for BrushNet). Please report standard deviations or confidence intervals across multiple seeds and run a paired significance test (e.g., Wilcoxon signed-rank) for the main CS/IR/AS/CMMD comparisons. The user study in Appendix A likewise reports single percentages from 40 participants on 20 images without confidence intervals; a paired preference test would strengthen the claim of user-level superiority.","section":"Sec. V-B2, Tables I-II, and Appendix A"}],"minor_comments":[{"comment":"The caption says 'when λ exceeds 0.8, the quality of the inpainted content begins to noticeably decline,' but the figure varies τ, not λ. This is likely a typo and should be corrected.","section":"Sec. V-C3, Fig. 9 caption"},{"comment":"The soft mask is written as cM_f = (1-τ)M_f + τ/HW, but HW is a scalar count while M_f is a vector; the intended expression is presumably a constant vector with entries τ/HW applied to all positions. Please clarify the notation.","section":"Sec. IV-A, Eq. (7)"},{"comment":"The text says the baseline uses 'only the key's mean,' but Eq. (10) defines eK as preserving unmasked keys and replacing only masked keys. It should be explicitly stated whether the baseline in Fig. 3 replaces all keys with the mean or only masked keys, since this changes the interpretation of the comparison.","section":"Sec. IV-B, Eq. (10) and Fig. 3"},{"comment":"The layer ranges are not fully specified: SAMS is applied to layers 2-6 and MAKVS to the final 8 layers, but it is unclear whether 'layers' refer to encoder/decoder blocks, how heads are treated, and whether the ranges overlap. Please give a complete per-block configuration for the U-Net.","section":"Sec. V-A1 and Sec. IV-C"},{"comment":"The caption reads 'Input w/o SAMS HarmonPaint w/o w/o MAKVS,' which appears to contain a duplicated 'w/o' and is confusing. The intended labels are likely 'Input,' 'w/o SAMS,' 'HarmonPaint,' and 'w/o MAKVS.'","section":"Fig. 6 caption"},{"comment":"FLUX is only compared in the appendix and not listed among the main competitors in Section V-A2. Given that the method is training-free, FLUX is a relevant baseline and should at least be cited and positioned in the main experimental section.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper's scope and topic are appropriate for a computer vision venue, and the central idea is interesting. The major concerns above are technical and experimental; I do not see evidence of citation manipulation or other ethical issues. I would be willing to look at a revised version that resolves the equation inconsistencies and adds non-StyleID evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about training-free diffusion editing. The core recipe is new: masking obj-bg quadrants in self-attention (SAMS) and injecting mean K/V from unmasked regions via a concatenation trick (MAKVS), plus an attention-steering loss (Ls). The paper shows real engineering skill and the qualitative figures are convincing. The ablations are thorough: every component, hyperparameter sweeps, mask types, partial inpainting, comparison with FLUX, and a user study. The authors are also honest about the 90%-mask failure mode.\n\nNow the soft spots. First, the SAMS math: masking is applied after softmax without renormalizing rows. For a masked token, the row of the attention matrix now sums to the original attention mass inside the mask, which is less than 1 whenever obj-bg attention was nonzero. That means the operation is not just selective masking; it also scales down the features of the masked region. The paper attributes structural fidelity to semantic separation, but the ablations don't disentangle scaling from masking. Add one sentence on renormalization and a control experiment, and the mechanism claim becomes much cleaner. Second, the benchmark circularity is real: both test sets are generated by StyleID, and MAKVS is essentially an inference-time relative of StyleID's K/V injection. The stylization is in-distribution for exactly the mechanism being tested. A neutral benchmark (other style transfer pipelines, or real artistic images) would make the headline claim much stronger. Third, the paper gives no code, no seeds, no error bars, so the quantitative gains (CLIP score, IR, AS) are hard to interpret. Fourth, the Ls guidance presumably requires backprop through the U-Net at inference, but the paper never says how the gradient is applied to the latent. That is a reproducibility gap.\n\nNone of this is fatal. The central idea is coherent, the implementation details are mostly clear, and the limitations section is actually a limitation section. With renormalization analysis, a neutral benchmark, error bars, and code release, this would be a solid publishable unit. As is, it deserves referee time: the core contribution is interesting enough that a capable reviewer can push it into shape.","headline":"A genuinely training-free inpainting recipe (self-attention quadrant masking plus key/value statistics injection) with strong qualitative results, but the evaluation is too self-referential (StyleID-made benchmarks) and the masking mechanism is under-specified without row renormalization.","tokens_in":16910,"tokens_out":3412,"would_cite":true,"duration_ms":41024,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By masking object-background attention and substituting mean key/value features, HarmonPaint produces structurally faithful and stylistically harmonized inpaints with no training.","keywords":["text-guided image inpainting","training-free inpainting","self-attention masking","style transfer","diffusion models","image harmonization","attention steer loss"],"falsifier":"Take an image whose unmasked region contains two distinctly styled halves, inpaint an object in the middle with HarmonPaint, and inspect whether the generated object adopts a coherent blend of both styles or collapses to an average that matches neither; the second outcome would falsify the claim that mean key/value features carry the surrounding style.","tokens_in":1724,"feed_emoji":"🎨","tokens_out":1970,"duration_ms":87781,"temperature":0.7,"pith_summary":"HarmonPaint is a training-free modification of the Stable Diffusion inpainting model, and the paper's central claim is that it can fill masked regions with content that is both structurally faithful and stylistically consistent with the rest of the image. Existing text-guided inpainting methods either retrain or fine-tune specialized heads, yet still produce unnatural boundaries or style mismatches when the image has a strong artistic style. HarmonPaint instead intervenes directly in the attention blocks: it masks object-background interactions in self-attention maps so the masked region develops its own principal components, and it copies mean key and value features from the unmasked region into the masked region so the new content adopts the surrounding style. The paper reports that this training-free recipe beats fine-tuned inpainting models and a large generative baseline on stylized benchmarks and in a user study, and that it degrades only when the mask covers more than 90 percent of the image. If true, the result shows that attention-level control alone, without new parameters, is enough for harmonized text-guided inpainting across styles.","feed_headline":"Attention tweaks make training-free inpainting match trained models","feed_subtitle":"Two attention edits keep inpainted objects structurally sound and stylistically consistent, no retraining required.","key_machinery":"The central object is the self-attention map of the diffusion U-Net. HarmonPaint partitions this map into object-object, background-background, and object-background blocks using the inpainting mask, zeroes the object-background block with a soft version of the mask to avoid hard boundaries, and feeds the reweighted map back into the encoder so masked and unmasked regions occupy distinct principal components. The style-transfer mechanism is the Mask-Adjusted Key-Value Strategy: the mean of K and V computed over unmasked patches replaces the K and V of masked patches, with the original K concatenated alongside the replaced key as [K, lambda eK] to keep content and style in balance. The Attention Steer Loss, built from cross-attention maps at resolutions 16 and 32, pushes prompt-token attention to concentrate inside the mask, and the division strategy runs structural masking in the early denoising stage and style transfer in the late stage.","core_discovery":"On its own terms, the paper discovers that the failure of diffusion inpainting to harmonize with its background can be traced to two properties of self-attention, and both can be corrected without training. First, self-attention maps in the U-Net encoder treat masked and unmasked regions as one principal component, so background features leak into the inpainted object; masking the object-background entries of the attention map separates the regions and preserves structure. Second, the key and value features of decoder self-attention carry style, so replacing the masked region's K and V with the mean of the unmasked region's K and V transfers the image's style into the generated content; concatenating the original and replaced keys with a strength parameter lambda prevents the style transfer from destroying object shape. An auxiliary cross-attention loss concentrates each prompt token's attention inside the mask, and a two-stage denoising schedule applies structural control early and style control late. The paper's conclusion is that these mechanisms together produce inpainting that is text-aligned, structurally coherent, and stylistically unified across oil-painting, sketch, and natural-image inputs, and that the approach outperforms both trained inpainting heads and larger generative models on the metrics and user preferences reported.","pith_inferences":["Beyond the paper: the same mean key/value substitution should apply to other localized edits, such as object replacement, relighting, or background swap, wherever the unmasked area defines the target style; the paper only demonstrates inpainting.","Beyond the paper: the global mean assumes one dominant style in the unmasked region, so a natural extension is region-adaptive means, for example clustering unmasked K/V features, for images containing several distinct style zones.","Beyond the paper: because the benchmark images are themselves produced by an external style-transfer model, the reported harmony is measured against that model's notion of style; real artwork with independent style labels would be a stiffer test.","Beyond the paper: the concatenation trick means attention is computed against both the original and the style-averaged key in one softmax, so whether lambda should vary over denoising steps rather than staying fixed is an open question that could refine the style-content trade-off."],"forward_implications":["No retraining or fine-tuning is needed: the same frozen diffusion inpainting weights handle stylized images, natural images, segmentation masks, bounding-box masks, and loose user scribbles by changing attention maps alone.","The method's style transfer is controlled by a single strength parameter lambda; setting it to zero reverts the model to ordinary inpainting, and lowering it to 0.8 adapts the same pipeline to natural images.","Because the masked region becomes its own principal component in self-attention, the generated object follows the mask shape even when the mask carries no semantic outline, as shown with arbitrary butterfly-shaped masks.","The reported gains over fine-tuned inpainting heads and a large generative baseline, if they hold, imply that task-specific training data for style is not a prerequisite for harmonized inpainting.","The paper's own stated bound is that the method relies on the unmasked region as its style source; when more than 90 percent of the image is missing, the style signal becomes insufficient for harmonization."],"supporting_citations":[{"why":"Provides the pre-trained diffusion inpainting model whose self-attention and cross-attention blocks HarmonPaint modifies.","marker":"[6]"},{"why":"Is the source of the claim that K and V carry style, and is used to construct the stylized test sets.","marker":"[61]"},{"why":"Supports the claim that self-attention maps encode image layout, motivating the masking strategy.","marker":"[48]"},{"why":"Provides the second prior showing self-attention features control structure, used to justify structural fidelity via attention.","marker":"[49]"},{"why":"Supplies the logical attention-guidance idea that HarmonPaint extends into the Attention Steer Loss.","marker":"[57]"},{"why":"Provides the soft-label smoothing used to avoid hard mask boundaries in the self-attention masking strategy.","marker":"[56]"},{"why":"A fine-tuned dual-branch inpainting model that serves as a main trained baseline to beat.","marker":"[2]"},{"why":"A fine-tuned task-prompt inpainting model used as the strongest trained comparison.","marker":"[3]"},{"why":"The training-free latent-blending baseline whose spatial mismatch motivates the proposed approach.","marker":"[4]"}],"fun_headline_variants":["Attention mask edits harmonize diffusion inpainting, no training","Training-free inpainting fixed by two attention tweaks","Self-attention edits harmonize inpainting without retraining","Masked keys and attention masks align inpainted style","No-training inpainting harmonization via self-attention"],"cache_read_input_tokens":19072,"weakest_assumption_plain":"The load-bearing premise is that averaging the key and value features from the unpainted region captures the image's style well enough that copying those averages into the masked region transfers style without eroding object content; this is borrowed from a single prior result on style injection rather than established inside the paper.","fun_headline_variants_meta":{"raw":{"variants":["Attention mask edits harmonize diffusion inpainting, no training","Training-free inpainting fixed by two attention tweaks","Self-attention edits harmonize inpainting without retraining","Masked keys and attention masks align inpainted style","No-training inpainting harmonization via self-attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000775,"raw_usage":{"total_tokens":3412,"prompt_tokens":912,"completion_tokens":2500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":2428}},"tokens_in":528,"tokens_out":2500,"duration_ms":18078,"temperature":1.0,"reasoning_tokens":2428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:01:32.035695+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an image whose unmasked region contains two distinctly styled halves, inpaint an object in the middle with HarmonPaint, and inspect whether the generated object adopts a coherent blend of both styles or collapses to an average that matches neither; the second outcome would falsify the claim that mean key/value features carry the surrounding style.","supporting_citations":[{"cited_title":"Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer,","cited_arxiv_id":null,"evidence_quote":"Is the source of the claim that K and V carry style, and is used to construct the stylized test sets."},{"cited_title":"Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing,","cited_arxiv_id":null,"evidence_quote":"Supports the claim that self-attention maps encode image layout, motivating the masking strategy."},{"cited_title":"Plug-and-play diffusion features for text-driven image-to-image translation,","cited_arxiv_id":null,"evidence_quote":"Provides the second prior showing self-attention features control structure, used to justify structural fidelity via attention."},{"cited_title":"Predicated diffusion: Predicate logic- based attention guidance for text-to-image diffusion models,","cited_arxiv_id":null,"evidence_quote":"Supplies the logical attention-guidance idea that HarmonPaint extends into the Attention Steer Loss."},{"cited_title":"Rethinking the inception architecture for computer vision,","cited_arxiv_id":null,"evidence_quote":"Provides the soft-label smoothing used to avoid hard mask boundaries in the self-attention masking strategy."},{"cited_title":"Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion,","cited_arxiv_id":null,"evidence_quote":"A fine-tuned dual-branch inpainting model that serves as a main trained baseline to beat."},{"cited_title":"A task is worth one word: Learning with task prompts for high-quality versatile image inpainting,","cited_arxiv_id":null,"evidence_quote":"A fine-tuned task-prompt inpainting model used as the strongest trained comparison."},{"cited_title":"Blended latent diffusion,","cited_arxiv_id":null,"evidence_quote":"The training-free latent-blending baseline whose spatial mismatch motivates the proposed approach."}],"review_version":1}