REVIEW 3 major objections 5 minor 30 references
Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Refine-by-Align is a two-stage diffusion pipeline that repairs localized identity artifacts in generated images by matching the masked region to a reference image through cross-attention, with no test-time tuning.
desk verdict New task and benchmark worth engaging with, but the main quantitative claim is undermined by selecting alignment hyperparameters on the test benchmark itself. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the cross-attention map $A = \mathrm{softmax}(qk^T)$, with queries from the noisy generated image and keys and values from 256 DINOv2 reference tokens. The alignment algorithm forms $M^{t,l} = \sum_{i,j} M_a \circ A[i,j,:]$, turning the masked artifact region into a 2D score map over reference locations; after noise removal and largest-blob selection this becomes the correspondence map $M^*$. A grid search over timesteps and transformer layers on the benchmark fixes the operating point at $t=0$, layer $l=9$, so inference needs only one denoising step for alignment. The same U-Net then uses the cropped reference $I_r \otimes M^*$ as guidance, and the two training modes, alignment mode and refinement mode, share weights, which is what lets one model do both localization and repair.
What would settle it
On a held-out artifact/reference set not used to select $t$ and $l$, compare the aggregated cross-attention map at $t=0$, layer 9 with human-annotated reference regions; low mIoU on large pose or shape mismatches, together with no DINO-image gain over feeding the full reference, would refute the alignment claim.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that the cross-attention maps of a diffusion model with a vision-encoder replacement carry enough spatial correspondence to localize a free-form artifact region inside a reference image, and that using only the localized reference region as guidance repairs the artifact while preserving identity. The paper implements this as Refine-by-Align, whose alignment stage aggregates attention maps over the masked pixels to produce a correspondence map, and whose refinement stage uses that map to crop the reference and drive the same weight-shared diffusion model to inpaint the masked area. On GenArtifactBench, the reported CLIP-image and DINO-image similarities are higher than for Paint-by-Example, ObjectStitch, AnyDoor, PAL, Cross-Image Attention, and MimicBrush, and a user study shows higher preference for both identity preservation and realism. Because the pipeline runs one forward pass at inference time, it adds no per-example optimization to the base generative model.
Load-bearing premise
The load-bearing premise is that at a single fixed diffusion timestep and transformer layer, aggregating the cross-attention map over the artifact mask always lands on the corresponding reference region, regardless of artifact shape, object category, or pose difference.
Editorial extensions
If this is right
- Any existing image generation model that produces localized identity artifacts can be paired with Refine-by-Align as a post-processing repair stage, without retraining or fine-tuning the generator.
- Users only need to mark the artifact with a free-form mask; the system then chooses the reference region automatically, so no manual correspondence is required.
- Because alignment selects a local reference patch instead of the whole object, tiny details such as logos, text, and textures are transferred more faithfully than with full-object guidance.
- The shared-weight design means the same checkpoint performs both localization and refinement, keeping the pipeline compact and reducing memory overhead compared with separate matching and inpainting models.
- GenArtifactBench supplies human-annotated artifact masks and reference correspondences, giving future artifact-refinement methods a common evaluation set.
Reading between the lines
- A natural next step is to test the same attention-aggregation recipe on other encoder-based diffusion models, since nothing in the algorithm is specific to the particular U-Net beyond the availability of cross-attention maps; if it transfers, alignment could become a generic module for reference-based editing.
- The paper selects the inference timestep and layer by grid search on the evaluation benchmark itself; a stronger validation of generality would hold out artifact categories or source models when choosing $t$ and $l$.
- Because the correspondence map is built from 16x16 reference tokens, very small logos or fine text may exceed the alignment resolution; using higher-resolution visual tokens or multi-scale attention could remove that ceiling.
- The paper does not automate artifact detection, but its benchmark masks could be used to train a detector that proposes masks automatically, turning the interactive pipeline into a fully automatic repair system.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Refine-by-Align proposes a two-stage, reference-guided pipeline for repairing localized artifacts in images produced by generative models. Given a generated image, a free-form artifact mask, and a reference image, the alignment stage uses cross-attention maps of a DINOv2-conditioned diffusion model to localize the region of the reference corresponding to the artifact, and the refinement stage inpaints the artifact using the extracted local reference features. The two stages share a single network and are trained self-supervised on Pixabay and a newly collected MVObj dataset. The paper also introduces GenArtifactBench, consisting of 146 annotated artifact/reference pairs from DreamBooth, Zero123++, AnyDoor, and IDM-VTON, and reports comparisons against six baselines using CLIP/DINO metrics and a user study.
Significance. The task is new and practically relevant, and the model-agnostic, no-test-time-optimization design is attractive. The creation of a benchmark with dense human annotations is a useful contribution, as is the self-supervised training scheme that avoids per-instance tuning. The paper also demonstrates qualitative results across several downstream generation tasks. However, the quantitative evidence for the central claim is currently weakened by the use of the evaluation benchmark for alignment-hyperparameter selection, and by the absence of uncertainty estimates. If the reported gains survive a clean held-out evaluation, the work would be a solid contribution; at present the headline superiority is not fully established.
major comments (3)
- [Sec. 4.4, Algorithm 1, Table 1] The alignment stage's only free parameters, the diffusion timestep t and transformer layer l, are selected by a grid search on GenArtifactBench using the ground-truth correspondence masks Mgt (Algorithm 1, lines 7-8; Fig. 4; Sec. 4.4), and the same benchmark is used to report the final numbers in Tables 1 and 2. This is evaluation leakage: the configuration is chosen on the test set, the baselines receive no equivalent selection step, and the chosen t=0, l=9 is not even the grid-search optimum (t=24, l=9), so the final configuration is a speed-accuracy tradeoff validated entirely on the benchmark. Please re-run the evaluation with a held-out split or nested cross-validation, and clearly state that the fixed configuration is selected without access to the GenArtifactBench labels.
- [Tables 1 and 2] The quantitative claims in Table 1 are reported as point estimates without error bars, confidence intervals, or significance tests. In Table 2, the smallest identity-preference margin is 55.83% vs. 44.17% (against ObjectStitch), and the CLIP-T score in Table 1 is actually below PAL (25.4063 vs. 25.8906); without variance estimates or a preference test the claim of consistently superior performance is not statistically supported. Please report bootstrap confidence intervals for the metrics and exact per-row comparison counts for the user study.
- [Sec. 3.2 and Sec. 3.3] The paper states that spatial correspondence 'exists' and is 'proved' in Sec. 3.2, but the argument is an informal intuition about cross-attention aggregation; no formal statement or direct quantitative validation is provided for arbitrary free-form masks, object categories, and pose differences. Since the alignment stage is the key novel component and the output M* is the only mechanism that distinguishes the method from blind inpainting, please provide a direct alignment-accuracy evaluation (e.g., mIoU against human annotations) on data not used for configuration selection, in addition to the qualitative comparison with DIFT/DHF in Fig. 2.
minor comments (5)
- [Algorithm 1] Algorithm 1 includes the ground-truth mask Mgt as an input and computes mIoU; this is a validation-time grid-search procedure, not the inference-time algorithm described in Sec. 3.5. Please relabel it accordingly and state that Mgt is unavailable at inference.
- [Sec. 3.2 and Algorithm 1] The notation is inconsistent: E is R^{256×768} in Algorithm 1 but R^{256×d_phi} in Eq. (3), and the mask Ma is resized to d×d in Algorithm 1 but not in Eq. (5).
- [Sec. A.4] The post-processing pipeline (noise filter, peak detection, clustering) is described only verbally; the threshold and clustering hyperparameters should be reported for reproducibility.
- [Table 2] The relationship between the 240 comparisons per question and the 720 total votes is unclear; please state the number of votes per comparison and the number of comparisons per baseline row.
- [Sec. 4.4] The analysis in Figs. 4 and 7 is called an ablation, but because it is computed on the same benchmark as the final evaluation, it is better described as configuration selection; use a validation set for parameter analysis.
Circularity Check
No derivation-level circularity; the only concern is benchmark hyperparameter selection, which is evaluation leakage rather than a circular reduction.
full rationale
The claimed derivation chain is not circular in any of the enumerated senses. The alignment stage's correspondence map M* is computed by aggregating cross-attention maps (Eq. 5) with a fixed inference choice (t=0, l=9), and the refinement stage is trained with the standard denoising objective (Eq. 2) on Pixabay and MVObj; neither quantity is defined in terms of the final evaluation metrics or of the human-annotated ground-truth correspondence masks. Algorithm 1 does take Mgt as an input, but only during the grid search for the timestep/layer hyperparameters; at inference the grid search is skipped, so M* is not computed from Mgt. The use of ObjectStitch/IMPRINT as backbone architecture is architectural inheritance, not a load-bearing self-citation argument. Section 3.2 says the spatial correspondence is 'proven', but the text actually gives an intuitive argument based on cross-attention; this is an unsupported assumption and a correctness risk, not a circularity. The one substantive issue is that the same GenArtifactBench is used to select t and l (Sec. 4.4) and to report the final comparisons (Tables 1-2); this is test-set model selection / data snooping, which can inflate the reported margins and should be noted as an evaluation-protocol flaw. It is not, however, a circular derivation: the final identity and realism numbers are not mathematically forced by the hyperparameter selection, and the paper does not reduce any prediction to its own inputs by construction. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- diffusion timestep t for alignment =
0
- transformer layer l for alignment =
9
- post-processing parameters (noise filter threshold, clustering) =
not specified
assumptions (3)
- domain assumption Cross-attention maps between the noisy artifact latent and the DINOv2 reference tokens encode a spatial correspondence that a masked aggregation can localize.
- domain assumption The self-supervised training pairs (Pixabay with perturbations, MVObj pairs) are representative of artifacts produced by DreamBooth, Zero123++, AnyDoor, and IDM-VTON.
- domain assumption A single fixed layer/timestep choice generalizes across all test images.
Cite this review
Pith. "Pith review of Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment." pith.science (2026). https://pith.science/paper/VEHO3FFC
@misc{pith2026241200306,
author = {Pith},
title = {Pith review of: Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment},
year = {2026},
howpublished = {\url{https://pith.science/paper/VEHO3FFC}},
note = {Machine review of arXiv:2412.00306}
}
read the original abstract
Personalized image generation has emerged from the recent advancements in generative models. However, these generated personalized images often suffer from localized artifacts such as incorrect logos, reducing fidelity and fine-grained identity details of the generated results. Furthermore, there is little prior work tackling this problem. To help improve these identity details in the personalized image generation, we introduce a new task: reference-guided artifacts refinement. We present Refine-by-Align, a first-of-its-kind model that employs a diffusion-based framework to address this challenge. Our model consists of two stages: Alignment Stage and Refinement Stage, which share weights of a unified neural network model. Given a generated image, a masked artifact region, and a reference image, the alignment stage identifies and extracts the corresponding regional features in the reference, which are then used by the refinement stage to fix the artifacts. Our model-agnostic pipeline requires no test-time tuning or optimization. It automatically enhances image fidelity and reference identity in the generated image, generalizing well to existing models on various tasks including but not limited to customization, generative compositing, view synthesis, and virtual try-on. Extensive experiments and comparisons demonstrate that our pipeline greatly pushes the boundary of fine details in the image synthesis models.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[2]
We train the model with a batch size of 192 and drop the image embedding at a rate of 0.1
MVObj, a dataset of 51k paired images. We train the model with a batch size of 192 and drop the image embedding at a rate of 0.1. The learning rate of the MLP connecting DINOv2 and U-Net is 4 × 10−5, and the U-Net has a learning rate of 1 × 10−5. The model is trained for more than 45 epochs on 8 NVIDIA A100 GPUs. A.4 P OST-PROCESSING OF THE CORRESPONDENCE...
work page 2023
-
[7]
Vision transformers need registers
Timoth´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588,
-
[8]
11 Sooyeon Go, Kyungmook Choi, Minjung Shin, and Youngjung Uh
URL https://arxiv.org/abs/2208.01618. 11 Sooyeon Go, Kyungmook Choi, Minjung Shin, and Youngjung Uh. Eye-for-an-eye: Appearance transfer with semantic correspondence in diffusion models. arXiv preprint arXiv:2406.07008 ,
-
[10]
Swapanything: Enabling arbitrary object swapping in personalized visual editing
Jing Gu, Yilin Wang, Nanxuan Zhao, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, and Xin Eric Wang. Swapanything: Enabling arbitrary object swapping in personalized visual editing. arXiv preprint arXiv:2404.05717,
-
[11]
Coho: Context-sensitive city-scale hierarchical urban layout generation
Liu He and Daniel Aliaga. Coho: Context-sensitive city-scale hierarchical urban layout generation. arXiv preprint arXiv:2407.11294,
-
[12]
Kubrick: Multimodal agent collaborations for synthetic video generation
Liu He, Yizhi Song, Hejun Huang, Daniel Aliaga, and Xin Zhou. Kubrick: Multimodal agent collaborations for synthetic video generation. arXiv preprint arXiv:2408.10453,
-
[13]
Prompt-to-prompt image editing with cross attention control
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626,
-
[15]
Multi-concept customization of text-to-image diffusion
12 Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1931–1941,
work page 1931
Show all 30 references
-
[16]
Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597,
-
[17]
Unihuman: A unified model for editing human images in the wild
Nannan Li, Qing Liu, Krishna Kumar Singh, Yilin Wang, Jianming Zhang, Bryan A Plummer, and Zhe Lin. Unihuman: A unified model for editing human images in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2039–2048,
-
[18]
Cliqueparcel: An approach for batching llm prompts that jointly optimizes efficiency and faithfulness
Jiayi Liu, Tinghan Yang, and Jennifer Neville. Cliqueparcel: An approach for batching llm prompts that jointly optimizes efficiency and faithfulness. arXiv preprint arXiv:2402.14833, 2024a. Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Che...
-
[19]
Diffusion hyperfeatures: Searching through time and space for semantic correspondence
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. In Advances in Neural Information Processing Systems, 2023a. Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and...
-
[20]
Kosmos- g: Generating images in context with multimodal large language models
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos- g: Generating images in context with multimodal large language models. arXiv preprint arXiv:2310.02992,
-
[21]
Sdxl: Improving latent diffusion models for high-resolution image synthesis
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952,
-
[22]
Orb: An efficient alternative to sift or surf
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In 2011 International conference on computer vision, pp. 2564–2571. Ieee,
2011
-
[24]
Instantbooth: Personalized text-to-image generation without test-time finetuning
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411, 2023a. Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and...
-
[25]
Empower- ing llms with pseudo-untrimmed videos for audio-visual temporal understanding, 2024b
Yunlong Tang, Daiki Shimada, Jing Bi, Mingqian Feng, Hang Hua, and Chenliang Xu. Empower- ing llms with pseudo-untrimmed videos for audio-visual temporal understanding, 2024b. URL https://arxiv.org/abs/2403.16276. Yunlong Tang, Gen Zhan, Li Yang, Yiting Liao, and Chenliang Xu....
-
[26]
Groundingbooth: Grounding text-to-image customization
Zhexiao Xiong, Wei Xiong, Jing Shi, He Zhang, Yizhi Song, and Nathan Jacobs. Groundingbooth: Grounding text-to-image customization. arXiv preprint arXiv:2409.08520,
-
[27]
Promptfix: You prompt and we fix the photo
Yongsheng Yu, Ziyun Zeng, Hang Hua, Jianlong Fu, and Jiebo Luo. Promptfix: You prompt and we fix the photo. arXiv preprint arXiv:2405.16785,
-
[28]
Llmexplainer: Large language model based bayesian inference for graph explanation generation
Jiaxing Zhang, Jiayi Liu, Dongsheng Luo, Jennifer Neville, and Hua Wei. Llmexplainer: Large language model based bayesian inference for graph explanation generation. arXiv preprint arXiv:2407.15351, 2024a. Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun...
-
[30]
and IDM-VTON Choi et al. (2024). Fig. 11 shows one example for each task. 18 Figure 10: User interface of the user study evaluating identity preservation. A.7 A DDITIONAL QUALITATIVE RESULTS We include more qualitative results in Fig
2024
-
[2006]
Synartifact: Classifying and alleviating artifacts in synthetic images via vision-language model.arXiv preprint arXiv:2402.18068,
Bin Cao, Jianhao Yuan, Yexin Liu, Jian Li, Shuyang Sun, Jing Liu, and Bo Zhao. Synartifact: Classifying and alleviating artifacts in synthetic images via vision-language model.arXiv preprint arXiv:2402.18068,
-
[2011]
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500...
-
[2014]
Photoswap: Personalized subject swapping in images
Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, et al. Photoswap: Personalized subject swapping in images. arXiv preprint arXiv:2305.18286,
-
[2018]
Cross- image attention for zero-shot appearance transfer
Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-Elor, and Daniel Cohen-Or. Cross- image attention for zero-shot appearance transfer. In ACM SIGGRAPH 2024 Conference Papers, pp. 1–12,
2024
-
[2020]
Finematch: Aspect-based fine-grained image and text mismatch detection and correction
Hang Hua, Jing Shi, Kushal Kafle, Simon Jenni, Daoan Zhang, John Collomosse, Scott Cohen, and Jiebo Luo. Finematch: Aspect-based fine-grained image and text mismatch detection and correction. arXiv preprint arXiv:2404.14715, 2024a. Hang Hua, Yunlong Tang, Chenliang Xu, and Jie...
-
[2021]
Improving dif- fusion models for virtual try-on
Yisol Choi, Sangkyung Kwak, Kyungmin Lee, Hyungwon Choi, and Jinwoo Shin. Improving dif- fusion models for virtual try-on. arXiv preprint arXiv:2403.05139,
-
[2022]
Surf: Speeded up robust features
Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In Computer Vision–ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13,
2006
-
[2023]
Zero-shot image editing with reference imitation
Xi Chen, Yutong Feng, Mengting Chen, Yiyang Wang, Shilong Zhang, Yu Liu, Yujun Shen, and Hengshuang Zhao. Zero-shot image editing with reference imitation. arXiv preprint arXiv:2406.07547,
-
[2024]
Anydoor: Zero- shot object-level image customization
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero- shot object-level image customization. arXiv preprint arXiv:2307.09481,
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.