Pith. sign in

REVIEW 3 major objections 6 minor 78 references

Pinco: Position-induced Consistent Adapter for Diffusion Transformer in Foreground-conditioned Inpainting

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that Pinco, a plug-and-play adapter costing about 11-13% of base-model parameters, outperforms existing foreground-conditioned inpainting methods by injecting subject features into self-attention and anchoring them with…

desk verdict Useful DiT inpainting adapter with a clean self-attention injection idea, but its headline OER wins are largely explained by extra depth/Sobel cues that baselines don't get, and the RoPE ablation is confounded by non-convergence. read the letter →

arxiv 2412.03812 v2 pith:LBZALSUA submitted 2024-12-05 cs.CV

Guangben Lu , Yuzhen Du , Zhimin Sun , Ran Yi , Yifan Qi , Yizhe Tang , Tianyi Wang , Lizhuang Ma
show 1 more author
Fangyuan Zou
This is my paper · ORCID
classification cs.CV
keywords foreground-conditionedinpaintingdiffusiontransformerself-attentioninjectiondecoupledimagefeatureextractionsharedpositionalembeddinganchorrotarysubjectshapepreservationtext-to-imageadapter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Pinco is a plug-and-play adapter for diffusion-transformer text-to-image models, aimed at inpainting the background behind a given foreground subject while leaving the subject's shape intact and following a text description. The paper claims that previous approaches either expand or distort the subject, drop text-specified objects, or need heavy side branches, and that all three problems can be addressed by injecting subject features into the self-attention layer, extracting semantic and shape information through separate encoders, and anchoring subject attention with positional embeddings. If the claim holds, large DiT models can do foreground-conditioned inpainting with roughly 11-13% of their parameters trainable and no retraining of the generator. The paper itself notes in Appendix H that very slender objects such as ropes or sticks and subjects photographed from unusual viewpoints remain hard cases.

What carries the argument

The machinery is a three-part adapter: (1) the Self-Consistent Adapter, which injects subject-aware cross-attention or MM-attention directly into the self-attention layer under two learnable gating parameters and zero-initialized tanh gating; (2) the Semantic-Shape Decoupled Image Feature Extractor, which reuses the frozen VAE encoder for semantic features and a small convolutional network over the mask, depth, and Sobel maps for shape features, then fuses them with an MLP; and (3) the Shared Positional Embedding Anchor, which bonds the backbone's rotary positional embedding to the subject feature keys so that the subject-aware attention map concentrates inside the subject region. The anchor is what turns scattered subject attention into localized attention, and the self-attention placement is what avoids text-versus-subject conflicts.

What would settle it

Retrain Pinco without the Shared Positional Embedding Anchor using the same data but with training extended until its OER and loss plateau, then compare attention maps and OER on the same 300-subject, 1,800-task benchmark; a converged no-anchor model matching Pinco would show the anchor is not necessary.

Watch

Extended reading notes

Core claim

The paper's central claim is that foreground-conditioned inpainting can be made consistent and cheap by keeping the base diffusion transformer frozen and training three small modules only. The Self-Consistent Adapter writes subject-aware attention into the self-attention path ($Z = \alpha \odot \mathrm{SelfAttention}(Q,K,V) + \beta \odot \mathrm{CrossAttention}(Q,K_{\mathrm{sub}},V_{\mathrm{sub}})$), so layout reasoning can see the subject without letting subject features fight the text cross-attention. The Decoupled Image Feature Extractor feeds semantic features from the backbone's VAE encoder and shape features from a convolutional net over mask, depth, and Sobel images, fused by an MLP. The Shared Positional Embedding Anchor reuses the backbone's rotary positional embedding on the subject keys so subject-aware attention decays away from the subject region. On Hunyuan-DiT and FLUX.1 the trained adapters report lower object-expansion ratios than ControlNet-style branches (e.g., Flux-Pinco 6.84% vs 22.30% OER) while using 11.37% and 12.56% of the base model's trainable parameters.

Load-bearing premise

The load-bearing premise is that the Shared Positional Embedding Anchor is what produces the reported gains, but the ablation used to show this compares with a no-anchor variant that never converged in the same training budget, so part of the gap could come from training instability.

Editorial extensions

If this is right

  • Pinco reports lower object-expansion ratios than ControlNet-style branches on both DiT backbones: HY-Pinco reaches 10.00% OER(BiRefNet) versus 11.95% for HY-ControlNet, and Flux-Pinco reaches 6.84% versus 22.30% for Flux ControlNet.
  • Because the base DiT stays frozen and only the adapter weights train, the method adds little latency and can be transferred to same-architecture backbones; the appendix demonstrates transfer to a community FLUX.1 finetune without extra training.
  • Injecting subject-aware attention into self-attention rather than combining it with text cross-attention output is claimed to reduce text-driven shape expansion and to converge faster during training.
  • The shared positional embedding anchor is claimed to make subject-aware attention focus on the subject region, suppressing the scattered attention that otherwise leaves shape and texture compromised.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same RoPE-anchoring trick could improve other DiT conditioning tasks that require attention to stay inside a reference region, such as object insertion, reference-based editing, or subject-driven generation.
  • Beyond the paper: because the shape branch fuses mask, depth, and Sobel channels, a per-channel ablation would test which cue actually drives the OER gains; the mechanism predicts mask and depth dominate.
  • Beyond the paper: the reported efficiency suggests that compact self-attention-injected adapters could scale to much larger DiTs, though that remains untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Pinco, a plug-and-play adapter for Diffusion Transformer (DiT) based text-to-image models, targeting foreground-conditioned inpainting: given a foreground subject image, a mask, depth and Sobel cues, and a text description, the model inpaints the background while preserving the subject's shape and aligning with the prompt. The method has three components: a Self-Consistent Adapter that injects subject features into the self-attention layers via an additional subject-aware attention branch; a Decoupled Image Feature Extractor that combines a frozen VAE encoder for semantic features with a small convolutional network for shape features from mask/depth/Sobel inputs; and a Shared Positional Embedding Anchor that reuses the base model's RoPE in the subject-aware attention to focus attention on the subject region. The method is instantiated on Hunyuan-DiT and FLUX.1 and evaluated on a self-collected test set of 300 objects with 1,800 generation tasks, using metrics for foreground consistency (OER with two segmentation models, LPIPS), text alignment (VQAScore, FV2Score), composition rationality (GPT-4o), image quality (FID), and trainable-parameter ratio, plus a user study and ablations.

Significance. If the central claim holds, Pinco is a practically valuable lightweight adapter: it reports roughly 11-13% trainable parameters relative to the base model, shows substantial OER improvements over ControlNet-style baselines on the same backbones, and demonstrates cross-backbone applicability. The architectural idea of injecting subject-conditioned attention into self-attention rather than cross-attention is a reasonable and potentially transferable design choice. The paper also includes a fairly broad evaluation: two DiT backbones, two off-the-shelf segmentation models for OER, multiple quality metrics, a user study, and explicit special-case tests. However, the evidence for superiority is weakened by an uncontrolled comparison: Pinco consumes extra geometric conditioning (depth and Sobel) that the baselines do not receive, and the paper's own ablation shows that removing this conditioning has a larger effect on the headline metric than the reported advantage over the closest baseline. The RoPE ablation is also not controlled because the w/o-RoPE variant fails to converge within the training budget. These issues are fixable but currently undermine the strength of the main claims.

major comments (3)
  1. [Sec. 4, Eq. (4); Sec. 5.2, Table 1; Sec. 5.3, Table 2] The main quantitative comparison is not controlled with respect to input conditioning. Pinco's input explicitly includes the subject depth map d and Sobel image s, which are fused through the convolutional shape branch in Eq. (4); none of the Table 1 baselines receive these extra geometric cues. Table 2 shows that removing this branch (Pinco-vae-only) raises OER(SAM2.1) from 11.51% to 36.11%, a 24.6-point degradation that is larger than the entire HY-Pinco versus HY-ControlNet gap (11.51 vs. 11.78) and a substantial fraction of the Flux-Pinco advantage (7.87 vs. 28.11). Consequently, the headline superiority may be attributable to the additional depth/edge priors rather than to the Self-Consistent Adapter or the adapter design. Please add a matched-input comparison: either supply equivalent depth/edge conditioning to the baselines where their architectures permit, or evaluate Pinco without d/s against the baselines, and report both settings.
  2. [Sec. 5.3, Fig. 8, Tab. 2] The ablation for the Shared Positional Embedding Anchor is not a controlled comparison. The paper states and Fig. 8 shows that Pinco-w/oRoPE fails to converge within the same epoch budget, producing images without the correct subject even at the final epoch. The large OER gap (290.44% vs. 11.51% for OER(SAM2.1)) is therefore not evidence that the RoPE mechanism causes the improvement; it may simply reflect training instability. Please train the w/o-RoPE variant to convergence, or match convergence state by a comparable training-loss or foreground-similarity criterion, before attributing the effect to the positional anchor.
  3. [Sec. 5.1, Table 1] All quantitative results are reported as point estimates without error bars, confidence intervals, or significance tests, even though each method is evaluated on 1,800 tasks (300 images, 3 prompts, 2 seeds) and several metrics use stochastic or model-based judges. Some headline gaps are small, e.g., OER(SAM2.1) of 11.51 for HY-Pinco versus 11.78 for HY-ControlNet, and it is unclear whether such differences are meaningful given the likely variance across prompts and seeds. Please provide per-item standard errors or bootstrap confidence intervals, accounting for clustering by subject image, at least for the primary metrics OER, VQAScore, and GPT-4o rationality.
minor comments (6)
  1. [Sec. 5.1, Evaluation Metrics] The FID evaluation is described only as 'evaluate FID on MSCOCO'; it is unclear which images form the reference distribution, whether FID is computed on full generated images, composited outputs, or background regions, and how the MSCOCO reference is matched to the inpainting test set. Please specify the protocol.
  2. [Sec. 5.1, Table 1] FV2Score is mentioned as measuring subject redundancy via Florence-2, but no definition, formula, or evaluation prompt is provided. Please define the metric precisely so readers can interpret the reported values.
  3. [Fig. 7] The attention-map visualization labels 'head No.7' and 'head No.11' without explaining which attention heads are shown, how the maps are aggregated across heads or layers, or what the color scale represents. Please clarify.
  4. [References] References [46] and [47] are the same paper (Peebles and Xie), and references [66] and [67] are also the same paper (Zhang et al.); these duplicates should be consolidated.
  5. [Sec. 5.2, User Study] The user study reports 31 participants and 40 comparisons each, but gives no details on participant recruitment, instructions, inter-rater agreement, or confidence intervals for the preference percentages in Fig. 5. Please add this information.
  6. [Appendix H, Limitations] The limitations section candidly notes failures on slender objects and unusual viewpoints; the main text could briefly acknowledge these failure modes near the qualitative results so the claims of 'superior performance' are appropriately scoped.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the method is benchmarked externally and the reported predictions do not reduce to fitted inputs or self-citations.

full rationale

Pinco's derivation is architectural and empirical rather than self-referential. The three proposed modules are trained with the standard diffusion denoising objective (Eq. 1) on a self-collected 88K-image training set and evaluated on a separately collected 300-item test set with 1,800 generation tasks per method, using off-the-shelf metrics such as OER, LPIPS, VQAScore, FV2Score, GPT-4o rationality, FID, and user preference. None of these metrics is a function of the adapter's trained weights, and the foreground-consistency conclusion is confirmed under two independent segmenters, BiRefNet and SAM2.1, so it is not tied to a single fitted segmenter. The depth and Sobel inputs enter the method by design through Eq. 4, but the claim that the Decoupled Image Feature Extractor helps is supported by the paper's own ablation in Table 2, which removes that branch; whether the main-table comparison is fully matched-input is a benchmarking-fairness concern, not a circularity. The unusual ablation note that Pinco-w/oRoPE fails to converge within the same epoch budget (Fig. 8) weakens the causal attribution of the Shared Positional Embedding Anchor, but it does not make the derivation equivalent to its inputs. The only self-citations, such as the authors' own ATA work in the reference list, are positioning citations and are not load-bearing; no uniqueness theorem or fitted prior is imported from them. Appendix H openly states limitations for slender objects and unusual viewpoints, which further indicates the evaluation is not constructed to force the claimed result. The unresolved U-ViT placeholder `[ ? ]` in Appendix B.1 is a completeness issue, not a circular one.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper is an empirical systems paper; it does not introduce free parameters in a derivation, but it does rely on hand-set hyperparameters and on several assumptions about the base models and metrics. No new physical or conceptual entities are introduced.

free parameters (3)
  • Number of ConvNet layers in shape extractor = 7
    Chosen by hand; the paper says the ConvNet consists of 7 convolutional layers and outputs from layers 1, 3, 5, 7 are used.
  • Block grouping for feature injection = 8 groups of 5 blocks
    Hunyuan-DiT's 40 blocks are partitioned into 8 groups; this hand-set grouping determines which feature scale each block receives.
  • Learning rate = 0.0001
    Fixed learning rate with 1K iteration warmup; not derived from data.
assumptions (5)
  • domain assumption Base T2I DiT models (Hunyuan-DiT, FLUX.1) are frozen and can generate high-quality images with text prompts.
    The adapter relies on the base model's pretrained generation ability; this is a stated premise in the method overview.
  • domain assumption Self-attention in diffusion models encodes spatial layout better than cross-attention for preserving structure.
    Based on prior work [37, 59], cited in Sec. 4.1 as the inspiration for the Self-Consistent Adapter.
  • domain assumption RoPE positional embedding from the base model, when added to subject keys, acts as an anchor to localize attention.
    The authors observe this empirically in Sec. 4.3 and Fig. 7, but provide no theoretical justification.
  • domain assumption The evaluation metrics (OER with SAM2.1/BiRefNet, VQAScore, FV2Score, GPT-4o rationality) measure the intended properties.
    The paper adopts these off-the-shelf metrics as ground truth, including an LLM judge for rationality, without validating them on this task.
  • domain assumption The collected 88K training images and 300 test images are representative of the foreground-conditioned inpainting distribution.
    The datasets are self-curated and not released, so representativeness and potential training/test overlap are unverifiable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pinco: Position-induced Consistent Adapter for Diffusion Transformer in Foreground-conditioned Inpainting." pith.science (2026). https://pith.science/paper/LBZALSUA

@misc{pith2026241203812,
  author       = {Pith},
  title        = {Pith review of: Pinco: Position-induced Consistent Adapter for Diffusion Transformer in Foreground-conditioned Inpainting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LBZALSUA}},
  note         = {Machine review of arXiv:2412.03812}
}
read the original abstract

Foreground-conditioned inpainting aims to seamlessly fill the background region of an image by utilizing the provided foreground subject and a text description. While existing T2I-based image inpainting methods can be applied to this task, they suffer from issues of subject shape expansion, distortion, or impaired ability to align with the text description, resulting in inconsistencies between the visual elements and the text description. To address these challenges, we propose Pinco, a plug-and-play foreground-conditioned inpainting adapter that generates high-quality backgrounds with good text alignment while effectively preserving the shape of the foreground subject. Firstly, we design a Self-Consistent Adapter that integrates the foreground subject features into the layout-related self-attention layer, which helps to alleviate conflicts between the text and subject features by ensuring that the model can effectively consider the foreground subject's characteristics while processing the overall image layout. Secondly, we design a Decoupled Image Feature Extraction method that employs distinct architectures to extract semantic and spatial features separately, significantly improving subject feature extraction and ensuring high-quality preservation of the subject's shape. Thirdly, to ensure precise utilization of the extracted features and to focus attention on the subject region, we introduce a Shared Positional Embedding Anchor, greatly improving the model's understanding of subject features and boosting training efficiency. Extensive experiments demonstrate that our method achieves superior performance and efficiency in foreground-conditioned inpainting.

Figures

Figures reproduced from arXiv: 2412.03812 by the authors.

Figure 1
Figure 1. Pinco generates high-quality images with rich and diverse backgrounds from given foreground subjects and text descriptions. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Foreground-conditioned inpainting results of three ex [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The framework of our Pinco, a plug-and-play inpainting adapter that can be seamlessly integrated with a text-to-image DiT model for consistent foreground-conditioned inpainting. Pinco consists of three modules: a Decoupled Feature Extractor used to extract subject feature, a Shared Positional Embedding Anchor used to ensure foreground attention, and a Self-Consistent Adapter injecting subject feature on the self-att… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Quantitative comparison of state-of-the-art inpainting methods and our [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: User Study. The images generated by Pinco received [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 8
Figure 8. Figure 8: The convergence process of Pinco with RoPE and with [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: The architecture of Flux-Pinco. For MM-DiT, we concatenate the latent and the subject features together to calculate the subject [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Method for obtaining multi-aspect ratio samples. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: GPT-4o prompt for assessment and its reply. Note that you need to specify the name of the subject in [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: The convergence analysis of Pinco, Pinco-Cross, and [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Some inference results under special cases. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: More cases generated by Pinco. Pinco supports the generation of high-quality images with different aspect ratios, while [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: More cases of the model training process. We provide a detailed demonstration of the training process of the Pinco, while [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Some cases in our user study. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: More qualitative comparisons between our Pinco and the state-of-the-art methods. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: More qualitative comparisons between our Pinco and the state-of-the-art methods. [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: More qualitative comparisons between our Pinco and the state-of-the-art methods. [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: GPT-4o rationality analysis results. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 38 canonical work pages

  1. [1]

    Accessed: 2023-10-04

    Adobe firefly, 2023. Accessed: 2023-10-04. 2, 7

    work page 2023

  2. [2]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18208–18218, 2022. 2, 3

    2022

  3. [3]

    Blended latent diffusion

    Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM transactions on graphics (TOG) , 42 (4):1–11, 2023. 2, 3

    work page 2023

  4. [4]

    Imagen 3

    Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brich- tova, Andrew Bunner, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, Zach Eaton-Rosen, et al. Imagen 3. arXiv preprint arXiv:2408.07009, 2024. 7

    arXiv 2024

  5. [5]

    Zoedepth: Zero-shot trans- fer by combining relative and metric depth

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M ¨uller. Zoedepth: Zero-shot trans- fer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023. 6

    arXiv 2023

  6. [6]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–10, 2023. 3

    work page 2023

  7. [7]

    Virtualmodel: Gener- ating object-id-retentive human-object interaction image by diffusion model for e-commerce marketing

    Binghui Chen, Chongyang Zhong, Wangmeng Xiang, Yifeng Geng, and Xuansong Xie. Virtualmodel: Gener- ating object-id-retentive human-object interaction image by diffusion model for e-commerce marketing. arXiv preprint arXiv:2405.09985, 2024. 6

    arXiv 2024

  8. [8]

    Disenbooth: Identity- preserving disentangled tuning for subject-driven text-to- image generation

    Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Identity- preserving disentangled tuning for subject-driven text-to- image generation. arXiv preprint arXiv:2305.03374, 2023. 3

    arXiv 2023

Show all 78 references
  1. [9]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 2, 3

    arXiv

  2. [10]

    Anydoor: Zero-shot object-level im- age customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level im- age customization. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6593–6602, 2024. 3

    2024

  3. [11]

    Audio-driven talking video frame restoration

    Harry Cheng, Yangyang Guo, Jianhua Yin, Haonan Chen, Jiafang Wang, and Liqiang Nie. Audio-driven talking video frame restoration. IEEE Transactions on Multimedia , 26: 4110–4122, 2024. 2

    2024

  4. [12]

    Latentpaint: Image inpainting in latent space with diffusion models

    Ciprian Corneanu, Raghudeep Gadde, and Aleix M Mar- tinez. Latentpaint: Image inpainting in latent space with diffusion models. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision , pages 4334–4343, 2024. 2, 3

    2024

  5. [13]

    Flux.1-dev controlnet inpainting beta,

    AliMama Creative. Flux.1-dev controlnet inpainting beta,

  6. [14]

    Ld-bfr: Vector-quantization-based face restoration model with latent diffusion enhancement

    Yuzhen Du, Teng Hu, Ran Yi, and Lizhuang Ma. Ld-bfr: Vector-quantization-based face restoration model with latent diffusion enhancement. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 2852–2860,

  7. [15]

    Exploring real&synthetic dataset and linear attention in image restoration

    Yuzhen Du, Teng Hu, Jiangning Zhang, Ran Yi Cheng- ming Xu, Xiaobin Hu, Kai Wu, Donghao Luo, Yabiao Wang, and Lizhuang Ma. Exploring real&synthetic dataset and linear attention in image restoration. arXiv preprint arXiv:2412.03814, 2024. 2

    2024 arXiv

  8. [16]

    Di- rect inversion: Optimization-free text-driven real image edit- ing with diffusion models

    Adham Elarabawy, Harish Kamath, and Samuel Denton. Di- rect inversion: Optimization-free text-driven real image edit- ing with diffusion models. arXiv preprint arXiv:2211.07825,

    arXiv

  9. [17]

    Salient object-aware background genera- tion using text-guided diffusion models

    Amir Erfan Eshratifar, Joao VB Soares, Kapil Thadani, Shaunak Mishra, Mikhail Kuznetsov, Yueh-Ning Ku, and Paloma De Juan. Salient object-aware background genera- tion using text-guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2024

  10. [18]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  11. [19]

    Scaling rectified flow trans- formers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yan- nik Marek, and Robin Rombach. Scaling rectified flow tr...

    2024

  12. [20]

    An image is worth one word: Personalizing text-to- image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patash- nik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to- image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022. 3

    2022 arXiv

  13. [21]

    Freeedit: Mask-free reference-based image editing with multi-modal instruction

    Runze He, Kai Ma, Linjiang Huang, Shaofei Huang, Jialin Gao, Xiaoming Wei, Jiao Dai, Jizhong Han, and Si Liu. Freeedit: Mask-free reference-based image editing with multi-modal instruction. arXiv preprint arXiv:2409.18071 ,

    arXiv

  14. [22]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 3

    2022 arXiv

  15. [23]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 7 9

    2017

  16. [24]

    Sara: High-efficient diffusion model fine-tuning with progressive sparse low-rank adapta- tion

    Teng Hu, Jiangning Zhang, Ran Yi, Hongrui Huang, Yabiao Wang, and Lizhuang Ma. Sara: High-efficient diffusion model fine-tuning with progressive sparse low-rank adapta- tion. In ICLR, 2024. 2

    2024

  17. [25]

    Hunyuancustom: A multimodal-driven architecture for customized video gener- ation

    Teng Hu, Zhentao Yu, Zhengguang Zhou, Sen Liang, Yuan Zhou, Qin Lin, and Qinglin Lu. Hunyuancustom: A multimodal-driven architecture for customized video gener- ation. arXiv preprint arXiv:2505.04512, 2025. 3

    2025 arXiv

  18. [26]

    Improving autoregressive visual generation with cluster- oriented token prediction

    Teng Hu, Jiangning Zhang, Ran Yi, Jieyu Weng, Yabiao Wang, Xianfang Zeng, Zhucun Xue, and Lizhuang Ma. Improving autoregressive visual generation with cluster- oriented token prediction. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 9351– 936...

    2025

  19. [27]

    Diffusion model-based image editing: A survey

    Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Shifeng Chen, and Liangliang Cao. Diffusion model-based image editing: A survey. arXiv preprint arXiv:2402.17525, 2024. 2

    2024 arXiv

  20. [28]

    Brushnet: A plug-and-play image inpaint- ing model with decomposed dual-branch diffusion

    Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and-play image inpaint- ing model with decomposed dual-branch diffusion. arXiv preprint arXiv:2403.06976, 2024. 2, 3, 4, 5, 6, 7, 13

    2024 arXiv

  21. [29]

    Flux-fill

    Black Forest Labs. Flux-fill. https://github.com/ black- forest- labs/flux/blob/main/docs/ fill.md, 2024. 3

    2024

  22. [30]

    Black Forest Labs. Flux. https://github.com/ black-forest-labs/flux, 2024. 3, 4, 6

    2024

  23. [31]

    Layerdiffusion: Layered controlled image editing with dif- fusion models

    Pengzhi Li, Qinxuan Huang, Yikang Ding, and Zhiheng Li. Layerdiffusion: Layered controlled image editing with dif- fusion models. In SIGGRAPH Asia 2023 Technical Commu- nications, pages 1–4. 2023. 7, 13

    2023

  24. [32]

    Gen- erative face completion

    Yijun Li, Sifei Liu, Jimei Yang, and Ming-Hsuan Yang. Gen- erative face completion. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 3911–3919, 2017. 2

    2017

  25. [33]

    Virtual try-on with pose-garment keypoints guided inpaint- ing

    Zhi Li, Pengfei Wei, Xiang Yin, Zejun Ma, and Alex C Kot. Virtual try-on with pose-garment keypoints guided inpaint- ing. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 22788–22797, 2023. 2

    2023

  26. [34]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding. arXiv preprint arXiv:2405.087...

    2024 arXiv

  27. [35]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  28. [36]

    Evaluating text-to-visual generation with image-to-text gen- eration

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text gen- eration. In European Conference on Computer Vision, pages 366–384. Springer, 2025. 7

    2025

  29. [37]

    Towards understanding cross and self-attention in stable diffusion for text-guided image editing

    Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, and Jun Huang. Towards understanding cross and self-attention in stable diffusion for text-guided image editing. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7817–7826, 2024. 4

    2024

  30. [38]

    Learning real facial concepts for independent deepfake detection

    Ming-Hui Liu, Harry Cheng, Tianyi Wang, Xin Luo, and Xin-Shun Xu. Learning real facial concepts for independent deepfake detection. arXiv preprint arXiv:2505.04460, 2025. 2

    2025 arXiv

  31. [39]

    Cones: Concept neurons in diffusion models for customized generation

    Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.05125, 2023. 3

    2023 arXiv

  32. [40]

    Cones 2: Customizable image synthesis with multi- ple subjects

    Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones 2: Customizable image synthesis with multi- ple subjects. In Proceedings of the 37th International Con- ference on Neural Information Processing Syste...

    2023

  33. [41]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022. 2

    2022

  34. [42]

    Taming diffusion model for exemplar-based image translation

    Hao Ma, Jingyuan Yang, and Hui Huang. Taming diffusion model for exemplar-based image translation. Computational Visual Media, 10(6):1031–1043, 2024. 2

    2024

  35. [43]

    Hd-painter: high-resolution and prompt-faithful text-guided image inpainting with diffusion models

    Hayk Manukyan, Andranik Sargsyan, Barsegh Atanyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Hd-painter: high-resolution and prompt-faithful text-guided image inpainting with diffusion models. arXiv preprint arXiv:2312.14091, 2023. 2, 3, 7, 13

    2023 arXiv

  36. [44]

    Null-text inversion for editing real im- ages using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real im- ages using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023. 3

    2023

  37. [45]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, 2024. 3

    2024

  38. [47]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  39. [48]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2, 3, 5, 7, 13

    2023 arXiv

  40. [49]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- 10 Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph F...

    2024 arXiv

  41. [50]

    High-resolution image syn- thesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2021. 3

    2021

  42. [51]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 3

    2022

  43. [52]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  44. [53]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  45. [54]

    Contrastive pseudo learning for open-world deepfake attribution

    Zhimin Sun, Shen Chen, Taiping Yao, Bangjie Yin, Ran Yi, Shouhong Ding, and Lizhuang Ma. Contrastive pseudo learning for open-world deepfake attribution. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 20882–20892, 2023. 2

    2023

  46. [55]

    Rethinking open-world deepfake attribution with multi-perspective sensory learning

    Zhimin Sun, Shen Chen, Taiping Yao, Ran Yi, Shouhong Ding, and Lizhuang Ma. Rethinking open-world deepfake attribution with multi-perspective sensory learning. Interna- tional Journal of Computer Vision, 133:628–651, 2024. 2

    2024

  47. [56]

    Ata: Adaptive transformation agent for text-guided subject- position variable background inpainting

    Yizhe Tang, Zhimin Sun, Yuzhen Du, Ran Yi, Guangben Lu, Teng Hu, Luying Li, Lizhuang Ma, and Fangyuan Zou. Ata: Adaptive transformation agent for text-guided subject- position variable background inpainting. In Proceedings of the Computer Vision and Pattern Recognition Confere...

    2025

  48. [57]

    Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis

    Kolors Team. Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis. arXiv preprint,

  49. [58]

    Pyramid-vae-gan: Transferring hierarchical latent variables for image inpainting

    Huiyuan Tian, Li Zhang, Shijian Li, Min Yao, and Gang Pan. Pyramid-vae-gan: Transferring hierarchical latent variables for image inpainting. Computational Visual Media , 9(4): 827–841, 2023. 2

    2023

  50. [59]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1921–1930, 2023. 4

    1921

  51. [60]

    Instantid: Zero-shot identity-preserving generation in seconds

    Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, and Yao Hu. Instantid: Zero-shot identity-preserving generation in seconds. arXiv preprint arXiv:2401.07519, 2024. 4

    2024 arXiv

  52. [61]

    Noise based deepfake de- tection via multi-head relative-interaction

    Tianyi Wang and Kam Pui Chow. Noise based deepfake de- tection via multi-head relative-interaction. In AAAI Confer- ence on Artificial Intelligence, pages 14548–14556, 2023. 2

    2023

  53. [62]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 481...

    2024

  54. [63]

    Magicremover: Tuning-free text-guided image inpainting with diffusion models

    Siyuan Yang, Lu Zhang, Liqian Ma, Yu Liu, JingJing Fu, and You He. Magicremover: Tuning-free text-guided image inpainting with diffusion models. arXiv preprint arXiv:2310.02848, 2023. 2, 3

    2023 arXiv

  55. [64]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  56. [65]

    Feditnet++: Few-shot editing of latent semantics in gan spaces with correlated attribute disentanglement

    Ran Yi, Teng Hu, Mengfei Xia, Yizhe Tang, and Yong-Jin Liu. Feditnet++: Few-shot editing of latent semantics in gan spaces with correlated attribute disentanglement. IEEE Transactions on Pattern Analysis and Machine Intelligence,

  57. [66]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2, 3, 4, 7, 13

    2023

  58. [67]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2, 3

    2023

  59. [68]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023. 4

    2023 arXiv

  60. [69]

    Bilateral refer- ence for high-resolution dichotomous image segmentation

    Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral refer- ence for high-resolution dichotomous image segmentation. arXiv preprint arXiv:2401.03407, 2024. 6

    2024 arXiv

  61. [70]

    Beta-tuned timestep diffu- sion model

    Tianyi Zheng, Peng-Tao Jiang, Ben Wan, Hao Zhang, Jin- wei Chen, Jia Wang, and Bo Li. Beta-tuned timestep diffu- sion model. In Computer Vision - ECCV 2024 - 18th Eu- ropean Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part III, pages 114–130. Springer, 2024. 3

    2024

  62. [71]

    A task is worth one word: Learning with task prompts for high-quality versatile image inpainting

    Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. arXiv preprint arXiv:2312.03594, 2023. 2, 3, 5, 7, 13 11 Appendix A. Overview In this supplementary material, we ...

    2023 arXiv

  63. [73]

    The compared baselines include: • SD1.5 backbone: ControlNet inpainting [66], HD- Painter [43], PowerPaint [71], and BrushNet-SD1.5 [28]; • SDXL backbone: SDXL inpainting [48], layerdif- fusion [31], BrushNet-SDXL [28], and Kolors- inpainting [57]; • DiT-based models: HY-Contr...

  64. [74]

    Determine whether the is placed in a physically impossible position, such as floating

    Object Placement Relationship: Check whether the spatial relationship between the and other objects in the image is reasonable and consistent with common placement methods in daily life. Determine whether the is placed in a physically impossible position, such as floating

  65. [75]

    Object Size Relationship: Assess whether the size proportions between the and other objects in the image are realistic, and whether there is any disproportion between the and surrounding objects

  66. [76]

    ## Rating Criteria: - 1 point: Obvious errors, inconsistent with the real world

    Physical Space Relationship: Consider whether the spatial distance between the and other objects in the image is reasonable, whether the perspective relationship conforms to the laws of the physical world, and whether there are any unreasonable aspects. ## Rating Criteria: - 1...

  67. [77]

    The bag is situated on the same level as the sitting dog, and it does not appear to be floating or placed in an unnatural position

    Object Placement Relationship: - The placement of the bag of dog food next to the dog on the ground appears realistic. The bag is situated on the same level as the sitting dog, and it does not appear to be floating or placed in an unnatural position. This is consistent with ho...

  68. [78]

    The dog food bag is reasonably large, similar to what would be expected for a product of that nature, and it does not appear disproportionately large or small compared to the dog

    Object Size Relationship: - The size proportion between the dog and the dog food bag seems realistic. The dog food bag is reasonably large, similar to what would be expected for a product of that nature, and it does not appear disproportionately large or small compared to the dog

  69. [79]

    A car on the street at night

    Physical Space Relationship: - The spatial distance and perspective between the dog and the dog food bag appear natural. The shadows and lighting are consistent with the positioning of the objects relative to one another. This suggests a reasonable perspective and physical spa...

  70. [2024]

    2, 3, 7, 13

    Accessed: 2023-10. 2, 3, 7, 13

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.