Pith. sign in

REVIEW 4 major objections 5 minor 58 references

PainterNet: Adaptive Image Inpainting with Actual-Token Attention and Diverse Mask Control

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PainterNet claims to beat prior inpainting models by steering diffusion attention onto masked regions with local prompts and an attention loss.

desk verdict PainterNet has a genuinely new attention loss and a useful local-prompt dataset, but the SOTA claim is not backed by the evaluation because the benchmark matches the method's training distribution and not the baselines'. read the letter →

arxiv 2412.01223 v1 pith:ELJBQP4K submitted 2024-12-02 cs.CV cs.AI

Ruichen Wang , Junliang Zhang , Qingsong Xie , Chen Chen , Haonan Lu This is my paper · ORCID
classification cs.CVcs.AI
keywords imageinpaintingdiffusionmodelstext-guidedcross-attentioncontrolactual-tokenattentionlossplug-and-playmaskgenerationPainterBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a plug-and-play inpainting plugin for Stable Diffusion can achieve better text-image consistency in masked regions than dedicated inpainting models, provided it is trained on local prompts and realistic, user-like masks. It proposes PainterNet, a dual-branch control network that keeps the full cross-attention structure of the base model, and adds an Actual-Token Attention Loss that pushes cross-attention maps onto the masked area so the generated pixels follow the local caption. A new training set, PainterData, provides local captions from multimodal LLMs and three mask styles (bounding boxes, irregular scribbles, and segmentation masks), while a new benchmark, PainterBench, evaluates on the same distribution. The paper reports that on this benchmark PainterNet beats BrushNet, ControlNet-Inpainting, HD-Painter, and Stable Diffusion inpainting on image reward, aesthetic score, global and local CLIP similarity, and grounding accuracy, and that the gains transfer to SDXL and to stylized downstream models.

What carries the argument

The core mechanism combines (1) a dual-branch control architecture, inherited from BrushNet but retaining cross-attention layers, where the trainable branch receives a 9-channel input (noisy latent, masked image latent, downsampled mask) and is fused into the frozen SD U-Net through zero convolutions at both layer control points and attention control points; and (2) the Actual-Token Attention Loss, $\mathcal{L}_{ATAL} = \frac{1}{N}\sum_{i=1}^{N}\| \frac{1}{L_S}\sum_{j \in S} A_{i,j} - m_i \|_2^2$, where $A_{i,j}$ is the cross-attention map of layer $i$ for text token $j$, $S$ is the set of actual text tokens (excluding start/end specials), and $m_i$ is the mask resized to that layer's attention map. This loss forces the model's cross-attention to concentrate on the masked region rather than on the whole image, which is what aligns generated pixels with the local prompt. The third component is the training-time mask mixer: for each sample, a random $k\in[0,1]$ selects a bounding-box mask (25%), an irregular finger-like scribble (50%), or the segmentation mask (25%), so the model learns to handle coarse, user-like inputs.

What would settle it

Rerun Table 1 on the original BrushBench (global prompts, segmentation masks) and on a held-out set of unscripted user-drawn scribbles; if PainterNet's margins over BrushNet and HD-Painter shrink or reverse, the state-of-the-art claim rests on benchmark overlap with the training distribution rather than on a general architectural advantage.

Watch

Extended reading notes

Core claim

PainterNet is a plug-and-play branch for Stable Diffusion-based inpainting that claims to fix two problems in existing control-based methods: global prompts that do not describe the masked region, and masks that do not match how users actually draw them. The paper replaces global captions with local prompts, keeps the full cross-attention layers of the trainable branch (unlike BrushNet, which removes them), and adds Attention Control Points plus an Actual-Token Attention Loss (ATAL) that drives the cross-attention maps of every layer toward the mask region. On the authors' PainterBench (built from PainterData with three mask types and MLLM-generated local prompts), PainterNet reports the best scores among BrushNet, ControlNet-Inpainting, HD-Painter, and Stable Diffusion inpainting on image reward, aesthetic score, global and local CLIP similarity, and Grounding DINO accuracy, with Local CLIP Sim of 22.67 on SD 1.5 and 23.06 on SDXL, and Gdino accuracy of 0.96 and 0.95. The ablation shows each component matters: the branch raises Local CLIP Sim from 22.40 to 22.53, ACP raises global CLIP Sim from 25.82 to 25.98, and ATAL lifts all key metrics.

Load-bearing premise

The results assume that evaluating every method on PainterNet's own style of local prompts and user-like masks is a fair comparison, even though the baselines were trained on global prompts and segmentation masks, so PainterBench may systematically favor PainterNet.

Editorial extensions

If this is right

  • PainterNet can be plugged into arbitrary Stable Diffusion-based models (the paper demonstrates anime, Van Gogh-style, and specific-role models) while keeping the base style intact.
  • Local prompts become the conditioning signal, so a user can describe just the masked object (e.g., 'a pink sweater') and the model aligns the inpainted pixels with that description rather than the full scene caption.
  • The ATAL loss compels cross-attention in every layer to concentrate on the mask, which the ablation ties to gains in all key metrics, especially local CLIP similarity and grounding accuracy.
  • Training on mixed mask shapes (box, irregular, segmentation) means users can draw coarse masks without following object contours, making the model usable with the masks people actually produce.
  • The same plugin trained on SD 1.5 transfers to SDXL, raising local CLIP similarity from 19.49 (SDXL-inpainting) to 23.06, suggesting the attention-control mechanism is backbone-agnostic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fair reading of the benchmark suggests part of the measured advantage may reflect that PainterBench matches PainterNet's training distribution (local prompts, mixed masks) while baselines trained on global prompts and segmentation masks face an out-of-distribution test; this distributional overlap, not just the architecture, could drive the reported margins.
  • The ATAL loss is generic enough to transfer beyond inpainting: any diffusion task that must respect a spatial region (object insertion, attribute editing, layout control) could reuse the same masked-attention objective.
  • The dataset recipe (MLLM captioning of cropped regions, short-caption post-processing, and CLIP-similarity filtering) is reusable for building localized caption datasets for other editing benchmarks, not just inpainting.
  • Varying the mask-mixing probabilities (fixed here at 25% box, 50% irregular, 25% segmentation) would be a natural next study; the paper does not test whether the reported robustness depends on that exact ratio.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PainterNet, a plug-and-play control branch for diffusion-based image inpainting. The method freezes the Stable Diffusion U-Net and inserts features from a trainable branch at both layer outputs and cross-attention outputs ("Attention Control Points"), using local textual prompts in both branches. An additional Actual-Token Attention Loss (ATAL) regularizes the branch's cross-attention maps toward the mask region. The authors also construct a new training set PainterData and a benchmark PainterBench, with local captions generated by an MLLM and diverse masks (bounding box, irregular scribbles, segmentation). Quantitative results on PainterBench (Table 1) and ablations (Table 2) are used to claim state-of-the-art performance on image quality and global/local text consistency.

Significance. The paper's architecture is a natural and reasonably clean extension of BrushNet: it retains cross-attention in the control branch, adds attention-level injection, and introduces a simple attention regularizer. The dataset and benchmark construction is described in enough detail in the supplementary (including Algorithm 1 and the CLIP threshold) to be reproducible in principle. I do not find a definitional circularity between the ATAL objective and the evaluation metrics. The problem is that the empirical support for the "surpasses state-of-the-art" claim is not yet convincing: PainterBench is built with the same prompt and mask generation pipeline as PainterData, while none of the baselines is adapted or fine-tuned for that setting, and the reported margins on the headline metrics are small and without error bars. The paper would be a solid contribution if the authors added a neutral benchmark evaluation, an adapted-baseline comparison, and basic statistical reporting.

major comments (4)
  1. [Section 4.1, Table 1] The state-of-the-art claim is confounded by the choice of benchmark. PainterBench is generated using the same mask-generation strategy (Section 3.4, Eq. (6); supplementary Section 6.2, Algorithm 1) and the same local-prompt MLLM pipeline (supplementary Section 6.1, Fig. 6) that are used to create PainterData. Baselines such as BrushNet and HD-Painter are trained on global prompts and segmentation-based masks (Sections 2.2 and 3.4). Evaluating them on PainterBench therefore gives PainterNet an in-distribution advantage while the baselines are out-of-distribution, and the metrics that most favor PainterNet (Local CLIP Sim, Gdino Acc) directly reward matching the local text in the masked region. The paper must additionally report results on BrushBench with the original global prompts and segmentation masks, or fine-tune the baselines on PainterData, or both. Without such a controlled comparison, the abstract's statement that PainterNet surpasses existing state-of-the-art models is not supported.
  2. [Section 4.2, Table 1] No error bars or significance tests are reported. The advantages on the headline metrics are small (Local CLIP Sim 22.67 vs 22.35; Gdino Acc 0.96 vs 0.94 on SD1.5), and the SDXL row actually shows a loss on IR and AS relative to SDXLI. The paper should report means and standard deviations over at least three runs (or bootstrap confidence intervals) and a paired test over the same masks and prompts, so the reader can judge whether the ranking is robust rather than noise.
  3. [Section 4.2] The evaluation protocol for the baselines is underspecified. The text states that inference follows the baselines' open-source code and recommended hyperparameters, but it never states whether the PainterBench local prompts were fed verbatim to all models, whether a global caption was also provided, or how bounding-box and irregular masks were converted for methods that expect segmentation masks or specific input formats. This must be stated explicitly, and the paper should also report each baseline's performance under its intended input format on a common benchmark. As written, the comparison is not well-defined.
  4. [Section 4.5, Table 2] The ablation does not isolate the contribution of ACP. Adding ACP alone decreases Local CLIP Sim (22.53 to 22.50) while increasing CLIP Sim, and the large jump to 22.67 appears only after ATAL is added. The paper does not report a variant with ATAL but without ACP, so the claim that both modules each contribute meaningfully is not demonstrated. Please add the missing ablation cell and, if the interaction is real, discuss it.
minor comments (5)
  1. [Section 3.2, 3.3] There are several typos, including "PatinerNet" and "BrtushNet" in Section 3.2 and "ACtual-Token Attention Loss" in Section 3.3; the manuscript should be proofread.
  2. [Section 4.2] The sentence "our PainterData utilized only 8% of the data from BrushData, amounting to 500,000 data" is unclear: does 8% correspond to 500,000 samples, and is PainterData a subset or a re-processed version of BrushData? Please clarify.
  3. [Eq. (5), Eq. (6), Supplementary 6.1] The hyperparameters β, the mask-selection thresholds in Eq. (6), and the CLIP threshold 0.2 in the supplementary are introduced without an ablation. Please report sensitivity or justify the chosen values.
  4. [Table 1] The CLIP Sim and Local CLIP Sim values (e.g., 26.12, 22.67) appear to be on a different scale from the usual CLIP cosine similarity in [0,1]; please state the exact formula and scaling used for these metrics.
  5. [General] The paper does not mention code or data release. Since the contributions include a dataset and a benchmark, providing them (or a link) would be important for reproducibility and for the community to use PainterBench.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PainterNet's training objective and evaluation metrics are not definitionally linked, though the author-built benchmark raises a separate fairness concern.

full rationale

The paper's central claim is that PainterNet surpasses state-of-the-art models on PainterBench. The derivation chain is: the model is trained with a combined loss L = Ldiff + beta*LATAL, where ATAL supervises the cross-attention maps to align with the resized mask region, and the model is then evaluated with Image Reward, Aesthetic Score, CLIP Similarity, Local CLIP Similarity, and Gdino Accuracy. There is no equation-level reduction of any reported metric to the training objective. LATAL forces attention maps toward the mask, but Local CLIP Sim measures the CLIP embedding similarity between the generated masked region and the local text prompt, and Gdino Acc measures whether the grounded detection box matches the prompt; neither is minimized directly by ATAL nor defined in terms of it. The fact that PainterBench was constructed with the same local-prompt and diverse-mask pipeline as PainterData is a legitimate evaluation-fairness concern because baseline methods were trained on global prompts and segmentation masks, but it is not circularity: the benchmark is not a subset of the training data, and the reported scores are not forced to equal any fitted parameter or training target by construction. No load-bearing self-citations or imported uniqueness claims appear in the paper. Accordingly, no circular step meets the evidentiary bar required here.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method does not introduce new physical entities or forces. It relies on standard pretrained diffusion components and adds data-processing and training hyperparameters. The key unstated assumptions are the quality of the automatically generated local prompts and the transferability of the attention regularization.

free parameters (3)
  • ATAL loss weight beta = 0.00001
    Set in Section 4.2 as a hyperparameter for training, not derived from theory.
  • CLIP similarity threshold for prompt filtering = 0.2
    Used in the supplementary Section 6.1 to filter generated local prompts; chosen by hand.
  • Mask selection probability thresholds = 0.25 and 0.75
    In Eq. (6), the random variable k selects among bounding box, irregular, and segmentation masks with fixed probability boundaries.
assumptions (3)
  • domain assumption Stable Diffusion pretrained weights are a suitable frozen backbone for inpainting.
    The method builds on SD 1.5 and SDXL, assuming their generative priors are sufficient for high-quality local editing.
  • domain assumption Local text prompts generated by multimodal LLMs (ShareGPT) and post-processed by ChatGLM provide accurate ground-truth descriptions of masked regions.
    The dataset construction in Section 3.4 relies on the correctness of these generated captions without human verification.
  • ad hoc to paper The ATAL loss, which pushes attention maps toward the mask, also improves final image-text consistency.
    This is an asserted link between an attention regularization and downstream metric performance; it is not proven but is empirically tested in the ablation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PainterNet: Adaptive Image Inpainting with Actual-Token Attention and Diverse Mask Control." pith.science (2026). https://pith.science/paper/ELJBQP4K

@misc{pith2026241201223,
  author       = {Pith},
  title        = {Pith review of: PainterNet: Adaptive Image Inpainting with Actual-Token Attention and Diverse Mask Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ELJBQP4K}},
  note         = {Machine review of arXiv:2412.01223}
}
read the original abstract

Recently, diffusion models have exhibited superior performance in the area of image inpainting. Inpainting methods based on diffusion models can usually generate realistic, high-quality image content for masked areas. However, due to the limitations of diffusion models, existing methods typically encounter problems in terms of semantic consistency between images and text, and the editing habits of users. To address these issues, we present PainterNet, a plugin that can be flexibly embedded into various diffusion models. To generate image content in the masked areas that highly aligns with the user input prompt, we proposed local prompt input, Attention Control Points (ACP), and Actual-Token Attention Loss (ATAL) to enhance the model's focus on local areas. Additionally, we redesigned the MASK generation algorithm in training and testing dataset to simulate the user's habit of applying MASK, and introduced a customized new training dataset, PainterData, and a benchmark dataset, PainterBench. Our extensive experimental analysis exhibits that PainterNet surpasses existing state-of-the-art models in key metrics including image quality and global/local text consistency.

Figures

Figures reproduced from arXiv: 2412.01223 by the authors.

Figure 1
Figure 1. Comparison of previous inpainting architectures and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of generation and datasets under local and global textual prompts. (a) Our method is able to generate correct results [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of our method. Our PainterNet introduces an additional branch that uses a hierarchical approach to gradually incor [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison of the performance of PainterNet and previous image drawing methods in various styles of drawing tasks: I, II for [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Generative effects of our PainterNet migration to other downstream models. Model I generates anime style outputs [ [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Generation of the local textual prompts. We first obtain the object location, crop it and input it into a multimodal large language [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Different mask shapes. Our mask generation strategy can generate diverse masks, which include segmentation-based [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: PainterBench overview. Our PainterBench ensures a uniform distribution of categories such as humans, animals, cartoons, and [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 24 canonical work pages

  1. [1]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18208–18218, 2022. 1, 2, 3

    work page 2022

  2. [2]

    Blended latent diffusion

    Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM transactions on graphics (TOG) , 42 (4):1–11, 2023. 1, 3, 8

    work page 2023

  3. [3]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 1

    2023

  4. [4]

    Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart- α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023. 1

    arXiv

  5. [5]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 2, 5, 6, 1

    arXiv 2023

  6. [6]

    Latentpaint: Image inpainting in latent space with diffusion models

    Ciprian Corneanu, Raghudeep Gadde, and Aleix M Mar- tinez. Latentpaint: Image inpainting in latent space with diffusion models. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision , pages 4334–4343, 2024. 1

    work page 2024

  7. [7]

    Van Gogh Diffusion Model

    Dallin Mackay. Van Gogh Diffusion Model. https: //huggingface.co/dallinmackay/Van- Gogh- diffusion, 2022. 2, 7, 8

    work page 2022

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 5

    2009

Show all 58 references
  1. [9]

    An image is worth one word: Personalizing text-to- image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patash- nik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to- image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022. 1

    2022 arXiv

  2. [10]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 63(11):139–144, 2020. 2

    2020

  3. [11]

    Counterfeit-v3.0

    gsdf. Counterfeit-v3.0. https://huggingface.co/ gsdf/Counterfeit-V3.0, 2024. 2, 7, 8

    2024

  4. [12]

    Prompt-to-prompt image editing with cross attention control.(2022)

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aber- man, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control.(2022). URL https://arxiv. org/abs/2208.01626, 2022. 1

    2022 arXiv

  5. [13]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 1

    2022 arXiv

  6. [14]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1, 2, 3

    2020

  7. [15]

    Introvae: Introspective variational autoencoders for photo- graphic image synthesis

    Huaibo Huang, Ran He, Zhenan Sun, Tieniu Tan, et al. Introvae: Introspective variational autoencoders for photo- graphic image synthesis. Advances in neural information processing systems, 31, 2018. 5

    2018

  8. [16]

    Brushnet: A plug-and-play image inpaint- ing model with decomposed dual-branch diffusion

    Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and-play image inpaint- ing model with decomposed dual-branch diffusion. arXiv preprint arXiv:2403.06976, 2024. 1, 2, 4, 5, 8

    2024 arXiv

  9. [17]

    Progressive growing of gans for improved qual- ity, stability, and variation.arXiv preprint arXiv:1710.10196,

    Tero Karras. Progressive growing of gans for improved qual- ity, stability, and variation.arXiv preprint arXiv:1710.10196,

    arXiv

  10. [18]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 4401–4410, 2019. 2

    2019

  11. [19]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection...

    1956

  12. [20]

    Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing

    Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Pro- cessing Systems, 36, 2024. 2

    2024

  13. [21]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  14. [22]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 6

    2023 arXiv

  15. [23]

    Deep learning face attributes in the wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015. 5

    2015

  16. [24]

    Spe- cialist diffusion: Plug-and-play sample-efficient fine-tuning of text-to-image diffusion models to learn any unseen style

    Haoming Lu, Hazarapet Tunanyan, Kai Wang, Shant Navasardyan, Zhangyang Wang, and Humphrey Shi. Spe- cialist diffusion: Plug-and-play sample-efficient fine-tuning of text-to-image diffusion models to learn any unseen style. In Proceedings of the IEEE/CVF Conference on Computer ...

    2023

  17. [25]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022. 1, 3

    2022

  18. [26]

    Hd-painter: high-resolution and prompt-faithful text-guided image inpainting with diffusion models

    Hayk Manukyan, Andranik Sargsyan, Barsegh Atanyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Hd-painter: high-resolution and prompt-faithful text-guided image inpainting with diffusion models. arXiv preprint arXiv:2312.14091, 2023. 4, 6, 8

    2023 arXiv

  19. [27]

    9 On distillation of guided diffusion models

    Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. 9 On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14297–14306, 2023. 3

    2023

  20. [28]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, 2024. 1

    2024

  21. [29]

    Gen- erating diverse structure for image inpainting with hierar- chical vq-vae

    Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li. Gen- erating diverse structure for image inpainting with hierar- chical vq-vae. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10775– 10784, 2021. 2

    2021

  22. [30]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 6, 8

    2023 arXiv

  23. [31]

    Deep learning-based image and video inpainting: A survey

    Weize Quan, Jiaxi Chen, Yanli Liu, Dong-Ming Yan, and Pe- ter Wonka. Deep learning-based image and video inpainting: A survey. International Journal of Computer Vision, 132(7): 2367–2400, 2024. 2

    2024

  24. [32]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  25. [33]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 2

    2020

  26. [34]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 1, 2

    2022 arXiv

  27. [35]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2, 3, 4, 6, 8

    2022

  28. [36]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  29. [37]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  30. [38]

    Mi-gan: A simple baseline for image in- painting on mobile devices

    Andranik Sargsyan, Shant Navasardyan, Xingqian Xu, and Humphrey Shi. Mi-gan: A simple baseline for image in- painting on mobile devices. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 7335– 7345, 2023. 2

    2023

  31. [39]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  32. [40]

    Server9. Ironman. https://civitai.com/models/ 509780/ironman, 2024. 2, 7, 8

    2024

  33. [41]

    Chatglm: A family of large language mod- els from glm-130b to glm-4 all tools

    GLM Team, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatglm: A family of large language mod- els from glm-130b to glm-4 all tools. arXiv e-prints, pages arXiv–2406, 2024. 5, 1, 2

    2024

  34. [42]

    High-fidelity pluralistic image completion with transform- ers

    Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transform- ers. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 4692–4701, 2021. 2

    2021

  35. [43]

    Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting

    Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont- Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting. In Proceedings of the IEEE/CVF c...

    2023

  36. [44]

    Godiva: Gen- erating open-domain videos from natural descriptions

    Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Gen- erating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806, 2021. 6

    2021 arXiv

  37. [45]

    Smartbrush: Text and shape guided object inpainting with diffusion model

    Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22428–22437, 2023. 1, 3, 4

    2023

  38. [46]

    Dreaminpainter: Text-guided subject-driven image inpaint- ing with diffusion models

    Shaoan Xie, Yang Zhao, Zhisheng Xiao, Kelvin CK Chan, Yandong Li, Yanwu Xu, Kun Zhang, and Tingbo Hou. Dreaminpainter: Text-guided subject-driven image inpaint- ing with diffusion models. arXiv preprint arXiv:2312.03771,

    arXiv

  39. [47]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation. Advances in Neural Information Pro- cessing Systems, 36, 2024. 6

    2024

  40. [48]

    Image completion with heterogeneously filtered spectral hints

    Xingqian Xu, Shant Navasardyan, Vahram Tadevosyan, An- dranik Sargsyan, Yadong Mu, and Humphrey Shi. Image completion with heterogeneously filtered spectral hints. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 4591–4601, 2023. 2

    2023

  41. [49]

    A review of image in- painting methods based on deep learning

    Zishan Xu, Xiaofeng Zhang, Wei Chen, Minda Yao, Jueting Liu, Tingting Xu, and Zehua Wang. A review of image in- painting methods based on deep learning. Applied Sciences, 13(20):11189, 2023. 1, 2

    2023

  42. [50]

    Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model

    Shiyuan Yang, Xiaodong Chen, and Jing Liao. Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model. In Proceedings of the 31st ACM 10 International Conference on Multimedia, pages 3190–3199,

  43. [51]

    Towards coherent image in- painting using denoising diffusion implicit models

    Guanhua Zhang, Jiabao Ji, Yang Zhang, Mo Yu, Tommi S Jaakkola, and Shiyu Chang. Towards coherent image in- painting using denoising diffusion implicit models. 2023. 1

    2023

  44. [52]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 1, 4, 6, 8

    2023

  45. [53]

    Large scale image comple- tion via co-modulated generative adversarial networks.arXiv preprint arXiv:2103.10428, 2021

    Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu. Large scale image comple- tion via co-modulated generative adversarial networks.arXiv preprint arXiv:2103.10428, 2021. 2

    2021 arXiv

  46. [54]

    Pluralistic image completion

    Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1438–1447, 2019. 2

    2019

  47. [55]

    Image inpainting with cascaded modulation gan and object-aware training

    Haitian Zheng, Zhe Lin, Jingwan Lu, Scott Cohen, Eli Shechtman, Connelly Barnes, Jianming Zhang, Ning Xu, Sohrab Amirghodsi, and Jiebo Luo. Image inpainting with cascaded modulation gan and object-aware training. In Eu- ropean Conference on Computer Vision , pages 277–296. Spr...

    2022

  48. [56]

    Construction of the dataset 6.1. Generation of the local textual prompts Due to the use of global text prompts in BrushData [16], detailed descriptions of masked regions cannot be provided, which may lead to inconsistencies between local generation and text prompts. To address...

  49. [57]

    to extract concise descriptions of the main objects, re- sulting in shorter local text prompts (as shown in II of Fig. 6). Finally, we use CLIP [32] to obtain the cosine similarity between the image and the shorter local prompt, and retain the prompt if the similarity exceeds ...

  50. [58]

    8, our PainterBench ensures a uniform distribution among various categories, including humans, animals, cartoons, as well as indoor and outdoor scenes

    PainterBench As illustrated in Fig. 8, our PainterBench ensures a uniform distribution among various categories, including humans, animals, cartoons, as well as indoor and outdoor scenes. This balanced allocation facilitates fair evaluations across different categories, enhanc...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.