{"id":"aa08d187-e05f-41eb-8f5d-070dfbf3906e","arxiv_id":"2501.02355","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A training-free plug-in that uses self-attention maps from Stable Diffusion inpainting to estimate and then enforce reference-target correspondences, improving fidelity of reference-based inpainting.","lead":"CorrFill is a training-free module that improves reference-based image inpainting by deriving correspondence constraints from the diffusion model's own attention maps and feeding them back into the denoising process. On RealEstate10K it raises PSNR by up to 3.84 dB over a strong baseline, but on MegaDepth the gains over the current state-of-the-art are marginal or zero.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains are selected results: hyperparameters in Supp. Table 1 were tuned on subsets of the same benchmark datasets (Supp. A.4), so the abstract's broad 'significantly enhances' claim is not yet supported as an unbiased estimate.","rationale":"The central empirical claim is that CorrFill significantly improves multiple diffusion-based baselines. The most load-bearing weakness is not the mechanism itself but the evidence for it: the authors disclose in Supp. A.4 that hyperparameters were selected by evaluating on subsets of the same datasets used for the final numbers, and Supp. Table 1 confirms per-baseline, per-dataset parameter choices. This makes the reported gains selected results rather than unbiased estimates. I considered whether the self-referential correspondence feedback (Eq. 1 and Eq. 5) is a more fundamental attack; it is a real limitation, and the paper's own failure cases and supplementary large-mask results show the method is not universal, but those caveats are explicitly acknowledged and do not refute the possibility of genuine gains on the evaluated distribution. The evaluation-protocol concern is concrete, disclosed, and directly testable. The reader's verdict is already CONDITIONAL, and this concern does not move it; it reinforces the need for code release and a fixed hyperparameter protocol, which are exactly the conditions stated in the verdict.","tokens_in":14319,"tokens_out":4445,"duration_ms":48599,"concrete_test":"Release code and rerun Table 1 with one fixed hyperparameter vector applied to all baselines and both datasets (e.g., the Side-by-side RealEstate10K settings), without subset-based tuning, reporting mean and standard deviation over multiple seeds. If the broad gains disappear or LeftRefill on MegaDepth remains at +0.00 dB, the headline claim is not supported; if meaningful gains persist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Supp. A.4 states that \"We selected the parameters by evaluating the subsets of our datasets,\" and Supp. Table 1 shows that the hyperparameters Stepa, Stepo, Wina, Wins, Stra, and Stro take different values for every baseline and every dataset. Table 1's means over 500 pairs are therefore best-case numbers chosen with knowledge of the test distribution, not estimates from a fixed method. The effect is material: LeftRefill, the strongest baseline, gains only +0.26 dB PSNR on RealEstate10K and +0.00 dB on MegaDepth, with essentially zero SSIM/LPIPS change, while IP-Adapter-Plus gains +3.84 dB on RealEstate10K. The abstract claims CorrFill \"significantly enhances the performance of multiple baseline diffusion-based methods, including state-of-the-art approaches,\" but no confidence intervals or significance tests are reported. Without code release and with per-dataset tuning, the observed gains could partly reflect hyperparameter search rather than the correspondence mechanism itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CorrFill, a training-free module for reference-based image inpainting with diffusion models. The method stitches the reference and target images side by side, aggregates target-to-reference self-attention maps across layers and timesteps of the denoising UNet into a matching map C_t, and extracts a correspondence field P_t via an argmax (Eq. 1). P_t is then used to guide the next denoising step through two mechanisms: attention masking, which boosts or hard-masks attention values near the predicted correspondence while suppressing dominant irrelevant tokens (Eqs. 3-4), and latent-tensor optimization, which applies a weighted BCE loss between per-layer target-to-reference attention maps and a one-hot encoding of P_t (Eq. 5). The process is cyclic: updated attention maps are accumulated back into C_t, refining P_t for the next step. Experiments in Table 1 report PSNR/SSIM/LPIPS gains over four diffusion-based baselines on 500-pair subsets of RealEstate10K and MegaDepth; Table 2 ablates the components; the supplementary reports a correspondence-accuracy comparison and a time analysis; the final section candidly lists failure modes (repetitive structures, large viewpoint changes, large masks).","tokens_in":14556,"tokens_out":12707,"duration_ms":120851,"significance":"The core idea--using the inpainting model's own self-attention as a shared domain for both correspondence estimation and guidance--is attractive and, if validated, would deliver a genuinely useful property: a plug-in that improves reference-target fidelity in reference-based inpainting without retraining or extra learnable parameters. Credit should go to the paper's real strengths: the method is architecture-compatible across several baselines; the ablation in Table 2 and the cyclic-versus-ablated correspondence diagnostics in Supp. B.2 are the right validation tools for the guidance loop; and the limitations (Sec. 5, Supp. C.2, Fig. 6) are reported honestly. However, the current evidence does not yet establish the headline claim: the largest gains are on baselines whose input representation is simultaneously changed, the state-of-the-art baseline (LeftRefill) gains only +0.26 dB PSNR on RealEstate10K and +0.00 dB on MegaDepth, and the reported numbers come from per-dataset hyperparameter selection on the evaluation subsets (Supp. A.4) without significance testing.","major_comments":[{"comment":"The headline improvements in Table 1 are computed with hyperparameters selected on subsets of the same benchmark datasets used for the reported means: Supp. A.4 states that 'We selected the parameters by evaluating the subsets of our datasets,' and Supp. Table 1 shows that the six parameters (Stepa, Stepo, Wina, Wins, Stra, Stro) take different values for every baseline and every dataset, with Stepa ranging from 5 to 50 and Stra set to 0 in one configuration. The 500-pair means in Table 1 are therefore best-case numbers from a per-dataset search rather than estimates from a fixed method, and no confidence intervals or significance tests are reported. Consequently, the abstract's claim that CorrFill 'significantly enhances the performance of multiple baseline diffusion-based methods, including state-of-the-art approaches' is not supported as an unbiased statement; in particular, the state-of-the-art baseline LeftRefill improves by only +0.26 dB PSNR on RealEstate10K and +0.00 dB on MegaDepth with essentially no SSIM change. I request a proper validation protocol: fix the parameters once (or tune on a separate validation split drawn from the same sources), report performance on a disjoint test set, and provide confidence intervals or paired tests for the gains.","section":"Supp. A.4 / Supp. Table 1 / Table 1"},{"comment":"The guidance loop is self-referential in a way that creates an error-lock-in risk: the correspondences P_{t+1} are computed in Eq. (1) as the argmax of the accumulated attention maps, and Eq. (5) then minimizes a BCE loss that coerces those very attention maps toward a one-hot encoding of P_{t+1}. When the initial argmax is wrong--which occurs in repetitive structures, large viewpoint changes, and large masks, all listed as failure modes in Sec. 5 and Supp. C.2--the loss sharpens attention around the wrong mode instead of correcting it, because the constraint target is derived from the same signal being constrained. Consistent with this risk, the ablation in Table 2 shows that attention masking alone degrades LeftRefill on RealEstate10K (26.71 to 26.45 dB) and only recovers after outlier filtering. The correspondence-accuracy analysis in Supp. B.2 is the right kind of evidence, but it is limited to RealEstate10K, it uses an argmax-based correctness metric that is closely tied to the guidance objective, and it does not diagnose the failure cases in Fig. 6. I suggest two additions: (i) an oracle bound in which P_{t+1} is replaced by the GIM pseudo-ground-truth correspondences, quantifying how much of the observed gain is lost to self-estimation error; and (ii) a plot of correspondence correctness over the 50 denoising steps for success versus failure examples, showing directly whether the cycle improves or locks in initial errors.","section":"Sec. 3.2 and Sec. 3.4, Eqs. (1) and (5)"},{"comment":"For Paint-by-Example and IP-Adapter-Plus, Sec. 4.2 states that the authors 'modify their inputs to match the stitched reference formulation and then apply our CorrFill,' so the reported gains (+1.54 dB and +3.84 dB on RealEstate10K) simultaneously change the input representation and add the guidance module. Because the corresponding baseline numbers come from the unstitched pipelines, the gains over these two baselines cannot be attributed to the correspondence guidance alone; part of the effect may come from the UNet directly seeing the reference through the stitched latent. No control column with stitched input and guidance disabled is provided for these baselines, and the implementation of the input modification for models whose released weights are not fine-tuned for the 9-channel inpainting input is not described in Supp. A.1. Please either provide matched-input baselines for Paint-by-Example and IP-Adapter-Plus or relegate these two rows to a supplementary comparison and base the plug-in claim on the stitched-formulation methods (Side-by-side and LeftRefill), where the comparison is clean.","section":"Sec. 4.2, Table 1"}],"minor_comments":[{"comment":"The function Norm(·) and the weighting in the weighted BCE are never defined, and the one-hot encoding E(·) requires a discretization decision for the sub-pixel smoothed correspondences P*_t; please specify these details, since Eq. (5) is the main driver of the large runtime increase reported in Supp. Table 2.","section":"Sec. 3.4, Eq. (5)"},{"comment":"Please clarify whether P_t stores token indices or continuous coordinates: Eq. (1) is an argmax over discrete tokens, whereas Sec. 3.3 adds fractional displacement fields D*_t (Eq. 2) and assigns them back to P_t, mixing discrete and continuous representations without comment.","section":"Sec. 3.2 and Sec. 3.3"},{"comment":"The PSNR values 27.39/27.34/27.25 dB for Ours/No acc./No cyc. differ from the LeftRefill RealEstate10K entry in Table 1 (26.97 dB); please state the configuration and subset used in that analysis so the reader can reconcile the two numbers.","section":"Supp. B.2"},{"comment":"For LeftRefill on MegaDepth the attention-mask boost value Stra is 0; please state what attention masking reduces to in that configuration (only the negative-infinity suppression, with no positive boost) and how the BCE loss remains active when the boost is zero.","section":"Supp. Table 1"},{"comment":"The dominant-token threshold of four tokens is claimed to be insensitive; support this claim with a one-line sensitivity check (for example, thresholds of 2, 4, and 8), since the threshold interacts with the multi-resolution aggregation across attention layers.","section":"Sec. 3.3"},{"comment":"Pair selection restricted to DreamSim distances between 0.1 and 0.2 on RealEstate10K bounds the task difficulty; please report the distributions of mask ratios and DreamSim distances in the released subset so that readers can gauge how challenging the benchmark is.","section":"Sec. 4.1"},{"comment":"The paper would benefit from a statement on code release, since the value of a plug-in method lies in the attention-substitution implementation across Diffusers pipelines; the project page currently promises only the dataset.","section":"Sec. 4"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper is well written and the idea is attractive as a training-free plug-in, and the supplementary contains the right kind of diagnostics (cyclic versus non-cyclic correspondence accuracy). My main concern is the evaluation protocol: the per-dataset, per-baseline parameter selection performed on the evaluation subsets (Supp. A.4) means the headline numbers are partially the product of the search, and for the strongest-baseline rows the gains are marginal (+0.26 dB PSNR on RealEstate10K and +0.00 dB on MegaDepth for LeftRefill). I recommend major revision with a held-out validation protocol and matched-input control experiments; if the authors can supply those along with fixed-parameter results, the paper would be a reasonable contribution. I would also ask for code release as part of the revision, given the difficulty of reimplementing the attention modification across the four baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CorrFill is a training-free plug-in that uses the diffusion model's own self-attention maps to estimate reference-target correspondences and then guides inpainting via attention masking and latent tensor optimization. The core idea is new in this combination, and the paper is clearly written. The ablation on RealEstate10K shows each component contributes, and the authors honestly report failure cases for repetitive structures and large viewpoint changes.\n\nThe main soft spot is the evaluation protocol. Supp A.4 states that hyperparameters were selected by evaluating on subsets of the same datasets used for the final numbers, and Supp Table 1 shows different parameters for every baseline and dataset. That makes the reported gains, especially the large jumps for IP-Adapter-Plus (+3.84 dB), partly a product of per-dataset tuning. On the strongest baseline, LeftRefill, the gain is only +0.26 dB on RealEstate10K and zero on MegaDepth. The abstract's 'significantly enhances multiple baselines' is stronger than what the evidence supports, especially without confidence intervals or significance tests.\n\nThe other concern is that the guidance loop is self-referential: correspondences come from the very attention maps that the method then coerces to match those correspondences. That could reinforce initial errors, though the authors' Figure 2 in the supplementary suggests the cyclic refinement does improve correspondence accuracy over time. The pseudo-ground-truth evaluation uses another learned matcher (GIM), so it is not an independent check.\n\nI would send this to peer review. The mechanism is plausible and the plug-in nature is useful. The authors should be asked to release code, fix the hyperparameters or report sensitivity across a validation set, and soften the claims to match the effect sizes. The paper is for researchers working on training-free guidance in diffusion models and reference-based inpainting; they will find the attention-based correspondence idea worth building on.","headline":"A useful training-free plug-in for diffusion inpainting, but the headline gains are inflated by per-dataset hyperparameter tuning and the strongest baseline gains are marginal.","tokens_in":15110,"tokens_out":2209,"would_cite":true,"duration_ms":20275,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CorrFill claims a training-free plug-in that uses self-attention-derived correspondences to guide diffusion inpainting, improving faithfulness to reference images and raising PSNR by up to 3.84 dB on tested baselines.","keywords":["reference-based image inpainting","diffusion models","correspondence guidance","training-free plug-in","self-attention","attention masking","latent tensor optimization","faithfulness"],"falsifier":"Take image pairs with known ground-truth geometry, for example rendered scenes with exact homographies, mask part of the target, and compare the method's estimated $P_t$ to the true flow; if precision in masked regions is no better than chance while PSNR gains disappear, the correspondence-guidance mechanism is not doing the claimed work.","tokens_in":14073,"feed_emoji":"🎨","tokens_out":7534,"duration_ms":69906,"temperature":0.7,"pith_summary":"The paper claims that diffusion-based reference-guided inpainting is unfaithful to the reference image because nothing in the pipeline enforces geometric correspondence between reference and damaged target. To fix this, it proposes CorrFill, a training-free plug-in that reads correspondences out of the self-attention maps the inpainting model already produces, then uses those correspondences to guide the remaining denoising steps through attention masking and latent tensor optimization. The authors show that plugging CorrFill into four different diffusion baselines improves PSNR, SSIM, and LPIPS on the RealEstate10K and MegaDepth datasets, with the largest PSNR gain being 3.84 dB on an image-adapter baseline. If the claim is right, CorrFill offers a general way to make existing reference-based inpainting models more faithful without retraining any weights.","feed_headline":"Self-attention maps guide diffusion inpainting, +3.84 dB","feed_subtitle":"A plug-in turns attention scores into reference-target geometry constraints, lifting PSNR without retraining.","key_machinery":"The load-bearing mechanism is the attention-consensus correspondence: the map $P_t$ computed as the argmax of the accumulated target-to-reference self-attention scores $C_t$ (Eq. 1), together with the filtering and smoothing steps that clean it. This correspondence is turned into guidance through attention masks $m_t$ that boost or suppress which reference tokens each target token may attend to, and through an objective $S(\\cdot)$ that optimizes the latent $z_t$ so its attention maps match the one-hot correspondence. The two roles feed each other: guidance shapes the denoising step, and the denoising step's fresh attention maps update the correspondence for the next step, which is the cycle that lets correspondence inside damaged regions improve without extra training.","core_discovery":"CorrFill's central claim is that a reference-based inpainting diffusion model can become more faithful if its own self-attention scores are converted into explicit geometric constraints between the reference and target images. Because the reference and target are stitched side by side, self-attention acts as a soft correspondence map; aggregating these maps over attention layers and timesteps and taking an argmax gives the correspondence $P_t$ (Eq. 1). The correspondences are then refined by removing dominant-token outliers and smoothing displacements, and are imposed on denoising by masking self-attention (Eqs. 3-4) and by optimizing the input latent $z_t$ with a weighted binary cross-entropy objective against the one-hot correspondence (Eq. 5). Since the updated attention maps feed back into the correspondence estimate, inpainting and correspondence estimation improve cyclically. The paper reports consistent gains over Paint-by-Example, IP-Adapter-Plus, Side-by-side, and LeftRefill, and attributes the largest gains to methods that were not trained specifically for reference-based inpainting.","pith_inferences":["An extension the paper does not explore is combining CorrFill with an external geometric prior or a pretrained matcher that initializes or verifies the attention-derived correspondences, which could reduce failures on repetitive structures and large viewpoint changes.","The same correspondence-guidance cycle could be applied to other stitched multi-image diffusion tasks, such as video frame completion or stereo inpainting, where geometric constraints are naturally stronger.","Because the guidance parameters were tuned on subsets of the two benchmark datasets, an independent re-evaluation on fresh pairs or a held-out dataset would show how much of the reported gain is parameter-free."],"forward_implications":["CorrFill is a drop-in module for any diffusion inpainting model whose self-attention operates over a stitched reference-target latent, requiring no retraining and no new parameters.","The reported results indicate that the largest gains occur for baselines not specifically trained for reference-based inpainting, so the module acts as a faithfulness corrector rather than a task-specific architecture.","The cyclic design implies that correspondence estimates in masked regions become more accurate as denoising proceeds, a property the supplementary supports by counting correct correspondences against pseudo-ground truth.","Correspondence smoothing, though small in average metrics, is credited with fixing specific wrong-geometry cases, so its role is case-dependent rather than uniform.","On wide-baseline or repetitive-structure scenes, the paper's own failure cases show that 2D correspondence guidance is not enough, so improvements there are expected to be smaller."],"supporting_citations":[{"why":"Supplies the pretrained Stable Diffusion inpainting U-Net that CorrFill modifies and whose self-attention maps carry the correspondence signal.","marker":"[19]"},{"why":"Provides LeftRefill and the Side-by-side stitched-reference formulation that CorrFill plugs into and improves.","marker":"[2]"},{"why":"Paint-by-Example is the CLIP-conditioned baseline that CorrFill must lift despite its information bottleneck.","marker":"[25]"},{"why":"IP-Adapter-Plus is the spatial-CLIP-token baseline where CorrFill reports its largest PSNR gain of 3.84 dB.","marker":"[26]"},{"why":"Shows that diffusion features and attention can serve as semantic correspondence, motivating the use of self-attention scores rather than external matchers.","marker":"[12]"},{"why":"Demonstrates zero-shot semantic correspondence from Stable Diffusion features, supporting the claim that correspondence can be estimated inside the inpainting model.","marker":"[28]"},{"why":"Supplies the attention-masking plus latent-optimization recipe that CorrFill adapts for reference-target correspondence.","marker":"[13]"},{"why":"Shows that training-free cross-attention guidance can steer a pretrained diffusion model, grounding the latent optimization component.","marker":"[4]"},{"why":"DreamSim is the metric used to select reference-target pairs for the evaluation datasets, so it shapes what the reported numbers measure.","marker":"[8]"},{"why":"Feature matching is used to generate masks over corresponding keypoints, determining the test conditions for reference-based inpainting.","marker":"[23]"}],"fun_headline_variants":["CorrFill: Training-free attention guidance for faithful inpainting","Self-attention correspondence lifts inpainting fidelity by 3.84 dB","Plug-in scores map reference geometry to guide inpainting","Reference-based inpainting gets a correspondence-guided boost","CorrFill: Attention maps keep inpainting true to reference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the argmax over accumulated self-attention scores identifies true reference-target correspondences, even inside the damaged regions, and that steering later denoising steps toward those correspondences corrects errors instead of amplifying them; if the attention scores are systematically wrong, the guidance reinforces the mistakes.","fun_headline_variants_meta":{"raw":{"variants":["CorrFill: Training-free attention guidance for faithful inpainting","Self-attention correspondence lifts inpainting fidelity by 3.84 dB","Plug-in scores map reference geometry to guide inpainting","Reference-based inpainting gets a correspondence-guided boost","CorrFill: Attention maps keep inpainting true to reference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1751,"prompt_tokens":927,"completion_tokens":824,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":736}},"tokens_in":543,"tokens_out":824,"duration_ms":7648,"temperature":1.0,"reasoning_tokens":736,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:39.721334+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take image pairs with known ground-truth geometry, for example rendered scenes with exact homographies, mask part of the target, and compare the method's estimated $P_t$ to the true flow; if precision in masked regions is no better than chance while PSNR gains disappear, the correspondence-guidance mechanism is not doing the claimed work.","supporting_citations":[{"cited_title":"Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained Stable Diffusion inpainting U-Net that CorrFill modifies and whose self-attention maps carry the correspondence signal."},{"cited_title":"Reference-based Painterly Inpainting via Diffusion: Crossing the Wild Reference Domain Gap","cited_arxiv_id":"2307.10584","evidence_quote":"Paint-by-Example is the CLIP-conditioned baseline that CorrFill must lift despite its information bottleneck."},{"cited_title":"Paint by example: Exemplar-based image editing with diffusion models","cited_arxiv_id":null,"evidence_quote":"IP-Adapter-Plus is the spatial-CLIP-token baseline where CorrFill reports its largest PSNR gain of 3.84 dB."},{"cited_title":"Megadepth: Learning single-view depth prediction from internet photos","cited_arxiv_id":null,"evidence_quote":"Shows that diffusion features and attention can serve as semantic correspondence, motivating the use of self-attention scores rather than external matchers."},{"cited_title":"Telling left from right: Identifying geometry-aware semantic correspondence","cited_arxiv_id":null,"evidence_quote":"Demonstrates zero-shot semantic correspondence from Stable Diffusion features, supporting the claim that correspondence can be estimated inside the inpainting model."},{"cited_title":"Diffusion hyperfeatures: Searching through time and space for semantic correspondence","cited_arxiv_id":null,"evidence_quote":"Supplies the attention-masking plus latent-optimization recipe that CorrFill adapts for reference-target correspondence."},{"cited_title":"Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Shows that training-free cross-attention guidance can steer a pretrained diffusion model, grounding the latent optimization component."},{"cited_title":"Training-free structured diffusion guidance for compositional text-to-image synthesis","cited_arxiv_id":null,"evidence_quote":"DreamSim is the metric used to select reference-target pairs for the evaluation datasets, so it shapes what the reported numbers measure."},{"cited_title":"Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, and Michael Rubinstein","cited_arxiv_id":null,"evidence_quote":"Feature matching is used to generate masks over corresponding keypoints, determining the test conditions for reference-based inpainting."}],"review_version":1}