Pith. sign in

REVIEW 4 major objections 6 minor 44 references

I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces text-guided multi-mask inpainting, in which multiple masked regions of one image are filled in a single diffusion pass, each region following its own text prompt, and shows that the prompts can be generated…

desk verdict New task and clean pipeline for multi-mask text-guided inpainting with automatic prompt generation; the WikiArt numbers are partly circular, but the DCI results and released code make it worth reviewing. read the letter →

arxiv 2411.19050 v2 pith:QST3JNIF submitted 2024-11-28 cs.CV

Nicola Fanelli , Gennaro Vessio , Giovanna Castellano This is my paper · ORCID
classification cs.CV
keywords text-guidedinpaintingmulti-maskpromptgenerationmultimodallargelanguagemodelsdiffusionrectifiedcross-attentionartworkrestorationparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Text-guided image inpainting usually fills one masked region from one prompt. This paper claims the task can be extended to many masks at once: a single pass through a diffusion model fills several regions simultaneously, each obeying its own prompt, without the usual prompt-blending failure mode. It further claims the prompts themselves can be produced automatically by a fine-tuned multimodal language model that looks only at the corrupted image, with each mask painted a distinct colour telling the model which prompt belongs where. If the claims hold, users of image-editing and art-restoration tools could describe several missing objects in natural language, or let the model suggest descriptions, and get all regions filled in one step. The supporting evidence comes from digitized artworks, where object-level annotations are manufactured automatically, and from a densely captioned photographic dataset.

What carries the argument

The load-bearing mechanism is rectified cross-attention (RCA), an attention-map surgery adapted from layout-to-image synthesis. In a text-guided diffusion model, the text conditions the image through cross-attention, giving an attention map $A = QK^\top/\sqrt{d}$ between text tokens and spatial positions. RCA constructs a layout tensor $L \in \{0,1\}^{T \times H \times W}$ by repeating each binary mask along the tokens of its corresponding prompt, then rectifies the attention map by $\widehat{A}_{k,i,j} = A_{k,i,j}$ where $L_{k,i,j}=1$ and $-\infty$ otherwise. Masks are modified so that unmasked regions attend to all tokens, and overlapping masks are stored as separate channels so intersections can attend to several prompts. This is what forces each generated object to stay in its own region while keeping the whole operation a single diffusion pass. The other half of the machinery is the prompt generator: a LLaVA multimodal language model fine-tuned with QLoRA, whose input is the corrupted image with each mask in a distinct random colour and whose output is the per-region captions wrapped in matching colour tags, so it learns the colour-to-region association from masked images alone.

What would settle it

Take a random sample of a few hundred masked regions in the artwork test set, obtain independent human-written object captions, and recompute prompt-generation accuracy and region-aligned CLIPSim-T2I against those human labels instead of the automatic annotations; a sharp drop would show the reported alignment is partly annotation bias. A second check: run the RCA-fine-tuned inpainter and per-mask repeated inpainting on a fixed set of human-verified prompts; if RCA no longer improves prompt-following, the central claim about region-locked attention fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a two-stage pipeline. A grounded captioning model first produces bounding boxes for salient objects, and a vision-language model writes a short object-level caption for each crop, creating training annotations for datasets that have none. A LLaVA-based prompt generator is then fine-tuned with QLoRA on examples where each mask is overlaid in a random colour and the expected answer is the corresponding captions wrapped in colour tags, so it learns to predict all hidden-object prompts from the corrupted image alone. For inpainting, Stable Diffusion's inpainting model is fine-tuned with LoRA and rectified cross-attention: when the per-region prompts are concatenated into one conditioning text, each prompt's tokens are forced to attend only to their own mask, unmasked regions attend to everything, and overlapping masks attend to both prompts. The paper reports that this RCA-fine-tuned model achieves the best region-to-prompt alignment in multi-mask settings, rivals per-mask repeated inpainting in quality while using a single pass, and, when fed the automatically generated prompts, clearly outperforms a generic 'high quality' prompt baseline.

Load-bearing premise

The load-bearing premise is that the automatically generated object labels and captions (from the grounded captioning model and the vision-language model) are accurate enough to serve as ground truth for both training and evaluation; the paper's own supplementary material concedes these automatically generated descriptions are 'inherently error-prone.'

Editorial extensions

If this is right

  • Multi-mask inpainting becomes a single-pass operation: the same number of diffusion steps as one-mask inpainting, with no repeated per-mask forward passes and no need to later stitch regions together.
  • The pipeline can run fully automatically: users give no text at all, and the fine-tuned prompt generator proposes plausible, varied per-region prompts from the corrupted image, with sampling temperature controlling diversity.
  • Datasets without object-level annotations become usable for this task, because the grounded captioning plus vision-language annotation stage manufactures the needed bounding boxes and captions, at the cost of some label noise.
  • RCA improves prompt-following over the base Stable Diffusion inpainting model, especially when several masks are present, and qualitative comparisons show less concept blending and more scene coherence than repeated per-mask inpainting.
  • The same pipeline transfers to photographic images, and the art-trained model can even be applied to photos, producing stylistically different but sometimes preferred completions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • (Editorial inference) Because both the prompt-generation metrics and the region-alignment metrics are computed against machine-made annotations, a human-labelled subset would be the natural stress test of whether the reported alignment is real or is partly an artefact of the annotation model's own bias.
  • (Editorial inference) The efficiency advantage of a single pass grows with the number of masks, since repeated inpainting costs one diffusion run per region; applications with many small regions, such as object-by-object editing or data augmentation, would benefit most.
  • (Editorial inference) The learned ability to guess what lies behind a mask from context could be reused for tasks beyond inpainting, such as 'what is missing?' reasoning about occluded or damaged scenes, though the paper does not explore that direction.
  • (Editorial inference) The acknowledged weakness on tiny or heavily overlapping masks points to a concrete next step: replace the binary mask layout with instance-aware or continuous region representations so that spatially ambiguous regions can still receive distinct prompts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces text-guided multi-mask inpainting, a new task in which several image regions are filled simultaneously under distinct text prompts. The proposed pipeline has two trainable components: (1) a LLaVA-based prompt generator, fine-tuned with QLoRA, that predicts multiple object-level prompts from a masked image alone, and (2) a Stable Diffusion 2.0 inpainting model fine-tuned with LoRA and an adapted rectified cross-attention (RCA) mechanism that forces each prompt to influence only its corresponding masked region. For the WikiArt artwork dataset, object-level supervision is obtained automatically via Kosmos-2 bounding boxes and LLaVA captions; for the Densely Captioned Images (DCI) dataset, human-annotated region captions are used. Experiments report prompt-generation quality (Accuracy, BLEU, ROUGE, CLIPSim) and inpainting quality (FID, LPIPS, PSNR, CLIP-IQA, CLIPSim-T2I), along with qualitative comparisons and a domain-transfer experiment. The central claim is that the full pipeline, including automatically generated prompts, outperforms baseline Stable Diffusion inpainting and performs comparably to repeatedly inpainting each mask individually, while requiring only a single diffusion pass.

Significance. If the empirical support is validated, the paper makes a useful contribution: it defines a practically relevant multi-mask inpainting task, combines parameter-efficient fine-tuning of an MLLM and a diffusion model, and adapts rectified cross-attention to handle overlapping masks. Strengths include the release of code, data, and trained models; a domain-transfer experiment that tests generalization; and the use of DCI's human-annotated captions as an independent source of supervision. The RCA adaptation is simple and clearly motivated, and the single-pass efficiency advantage over repeated inpainting is a concrete benefit. However, the primary evidence on the WikiArt dataset rests on annotations produced by the same model families that are being evaluated, and all quantitative results are point estimates without error bars or significance tests. These issues materially weaken the load-bearing claims about prompt-generation accuracy and prompt-following improvements, so the paper needs substantial empirical revision before the conclusions can be accepted.

major comments (4)
  1. [Sec. 3.1, Tables 1-3] The WikiArt evaluation is circular: the object-level 'ground truth' descriptions used to train and evaluate the prompt generator are produced by Kosmos-2 and LLaVA, the same model families used in the proposed pipeline. Section 4.1.3 defines Accuracy as checking whether the Kosmos-2 noun-chunk root appears in the generated prompt, and Table 3's CLIPSim-T2I uses the LLaVA captions as reference prompts. High scores can therefore reflect imitation of the annotator's biases rather than correct identification of hidden image content. The supplementary material (Fig. 6) itself states that 'automatically generated descriptions are inherently error-prone.' The only independent evidence, DCI with human captions, shows a prompt-generation Accuracy of only about 31-34% (Table 2) and a modest CLIPSim-T2I gain from RCA on DCI (+0.63, Table 4). I recommend either (a) re-evaluating the art-dataset claims with a human-verified gold standard on a sample of masked regions, or (b) demoting the art-dataset quantitative results and relying on DCI as the primary evidence for the pipeline's correctness.
  2. [All tables, Sec. 4.1.3] All quantitative results are reported as point estimates without error bars, confidence intervals, or significance tests. Several decisive comparisons are small: for example, Table 1 CLIPSim 24.65 vs. 24.79; Table 3 CLIPSim-T2I 25.81 vs. 26.72 and 26.95 vs. 26.72; Table 4 CLIPSim-T2I 24.85 vs. 25.48. Since the prompt generator uses stochastic sampling and the diffusion sampler uses randomized steps, these differences could easily fall within run-to-run variation. I request bootstrap confidence intervals over test examples (or over multiple seeds) and a paired significance test (e.g., Wilcoxon signed-rank) for the headline claims, especially the claim that RCA fine-tuning improves prompt-following relative to the repeated-inpainting baseline.
  3. [Sec. 4.1.3, Fig. 7] The prompt-generation Accuracy metric is too weak to support the paper's conclusions. It counts a match if the root of the Kosmos-2 noun chunk appears anywhere in the generated prompt. Given the long-tailed distribution of noun-chunk roots (Fig. 7d), a model that always outputs a frequent root such as 'person' would achieve non-trivial accuracy without correctly identifying the hidden entity. The paper does not report a most-frequent-class baseline, per-class accuracy, or a human evaluation of generated prompts. The fact that Accuracy on DCI, with human annotations, is only about 31-34% (Table 2) and is still considered the best result underscores the need for a more reliable evaluation. I recommend reporting the base rate of the most frequent roots, using DCI accuracy as the primary reliability measure, and adding a small human study of prompt plausibility.
  4. [Abstract, Sec. 4.2, Sec. 5] The paper claims that the proposed method produces 'more aesthetically pleasing qualitative outcomes compared to inpainting masks individually.' This claim rests entirely on the authors' qualitative inspection (Figs. 5, 9, 10 and supplementary) and is not supported by quantitative evidence: on the full test set in Table 3, the repeated-inpainting baseline achieves higher CLIPSim-T2I than the RCA model (26.95 vs. 26.72), and both have similar FID. A user preference study on at least a few dozen examples would substantiate the aesthetic claim, or the authors should explicitly restrict the claim to the efficiency advantage (single-pass inpainting) and to the qualitative examples.
minor comments (6)
  1. [Sec. 3.3, Eq. (3)] The construction of the layout L is described as 'repeating the binary masks Mi for each token in the corresponding prompt Yi'; please clarify how the token count is determined (e.g., which tokenizer is used) and whether special tokens such as punctuation or start/end tokens are included in the masking.
  2. [Sec. 4.1.2] The prompt-generation model is trained for a single epoch on the art dataset; please state how the final checkpoint was selected (e.g., by validation loss) and whether early stopping or a validation-based criterion was employed.
  3. [Table 3 caption] The caption refers to 'gray rows,' but in the manuscript text the rows are not visually distinguished; please ensure the final PDF clearly marks the repeated-inpainting rows in gray.
  4. [Sec. 4.2, Data Annotation Quality] The comparison of CLIPSim scores (0.32 and 0.28) with the LAION-5B threshold of 0.28 is not directly valid because the threshold used a different CLIP model and image-text distribution; please state the exact CLIP variant used and avoid over-interpreting the absolute threshold.
  5. [Eq. (1)] The notation I in the causal language-modeling loss is ambiguous; in the prompt-generation task, the model sees the corrupted image I_inp, not the original image. Please make this explicit in the equation or its surrounding text.
  6. [Fig. 4] Figure 4 is referenced in the text but its contents are not visible in the manuscript text provided; please ensure the figure, axis labels, and legend appear correctly in the submitted PDF.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: the WikiArt prompt-generation evaluation uses the same Kosmos-2/LLaVA annotations as both training targets and test references, so part of the reported 'quality' is self-agreement; the inpainting core and DCI results remain independent.

  1. fitted input called prediction [Section 3.1 (Data Annotation), Section 3.2 (Prompt Generation), Section 4.1.3 (Evaluation Metrics, Table 1)]
    "we have developed a data annotation pipeline that leverages MLLMs to produce these annotations ... we employ Kosmos-2 ... we prompt LLaV A to generate a detailed description of the object in the cropped image ... Accuracy was measured by checking if the root of the noun chunk from Kosmos-2 annotations appeared in the prompt."

    The prompt generator (LLaVA-7B) is fine-tuned on ground-truth answers A that concatenate LLaVA-13B/Kosmos-2 object descriptions (Section 3.2), and Table 1 evaluates it against the same annotation source: Kosmos-2 noun-chunk roots and LLaVA captions. Because LLaVA-13B received the Kosmos-2 noun chunk as a contextual hint when writing each caption, and the student model is trained to reproduce those captions, high BLEU/ROUGE/Accuracy can indicate imitation of the teacher's vocabulary rather than independent identification of the hidden object. The supplementary explicitly concedes that 'automatically generated descriptions are inherently error-prone,' so the WikiArt prompt-generation metrics are self-referential without a human-verified gold standard.

full rationale

The core multi-mask inpainting method is an adaptation of rectified cross-attention from FreestyleNet to inpainting, combined with QLoRA fine-tuning of LLaVA for prompt generation; these steps are externally sourced or empirically evaluated against real pixel targets and are not circular. The main circular element is confined to the WikiArt prompt-generation evaluation: the 'ground truth' descriptions used for training and for scoring are both produced by the same Kosmos-2 → LLaVA-13B annotation pipeline that the paper itself introduces, and the supplementary admits these descriptions are error-prone. The DCI dataset provides independent human-annotated captions and shows the same method achieving only modest absolute performance (e.g., 34.14% accuracy), which is consistent with the claim that the art-dataset numbers are inflated by self-consistency. The ArtGraph citation is from the same research group but is used only as an image source, not as load-bearing theoretical support, so it does not raise the score. Overall, the central contribution is not a derivation that reduces to its inputs, but part of the reported evidence on the art dataset is weakened by self-referential evaluation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The pipeline introduces no new physical or theoretical entities. The main load-bearing assumptions are the reliability of automatically generated annotations and of CLIP-based evaluation metrics. All hand-chosen thresholds (N=5, area caps, guidance weight) are design parameters that affect results but are not fitted to force a specific outcome.

free parameters (5)
  • Maximum number of masks N = 5
    Hand-chosen limit on the number of prompts per example; affects task difficulty and dataset distribution.
  • Bounding box area thresholds = 1% to 65% of image area
    Hand-chosen filters for keeping annotations; discards too small or too large objects.
  • Maximum total masked area = 65% of image area
    Hand-chosen cap to ensure enough context remains for inpainting.
  • Classifier-free guidance weight = 7.5
    Standard test-time setting for Stable Diffusion, chosen by hand.
  • Temperature for prompt sampling = 0.5 to 1
    Explored manually; higher temperature increases diversity at slight quality cost.
assumptions (5)
  • domain assumption Pre-trained models (Stable Diffusion 2.0 Inpainting, LLaVA, Kosmos-2, CLIP) perform as expected and their embeddings and attention mechanisms are reliable for this task.
    The whole pipeline relies on these models' pretrained capabilities, invoked in Section 3 and 4.
  • domain assumption Kosmos-2 and LLaVA produce sufficiently accurate object-level annotations to serve as ground truth.
    Section 3.1 uses these annotations for training and evaluation; the paper acknowledges error-proneness in supplementary Section A.
  • domain assumption CLIP similarity scores correlate with human judgment for image-text alignment and for region-prompt alignment.
    Used to justify annotation quality and prompt-following metrics; relies on cited work [12,23].
  • domain assumption Rectified cross-attention transfers from layout-to-image synthesis to inpainting without breaking the diffusion process.
    Section 3.3 adapts FreestyleNet's RCA; the paper provides no formal guarantee, only empirical results.
  • domain assumption Standard diffusion training objectives and LoRA fine-tuning preserve the base model's generative quality.
    Used throughout training in Section 4.1.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting." pith.science (2026). https://pith.science/paper/QST3JNIF

@misc{pith2026241119050,
  author       = {Pith},
  title        = {Pith review of: I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QST3JNIF}},
  note         = {Machine review of arXiv:2411.19050}
}
read the original abstract

Inpainting focuses on filling missing or corrupted regions of an image to blend seamlessly with its surrounding content and style. While conditional diffusion models have proven effective for text-guided inpainting, we introduce the novel task of multi-mask inpainting, where multiple regions are simultaneously inpainted using distinct prompts. Furthermore, we design a fine-tuning procedure for multimodal LLMs, such as LLaVA, to generate multi-mask prompts automatically using corrupted images as inputs. These models can generate helpful and detailed prompt suggestions for filling the masked regions. The generated prompts are then fed to Stable Diffusion, which is fine-tuned for the multi-mask inpainting problem using rectified cross-attention, enforcing prompts onto their designated regions for filling. Experiments on digitized paintings from WikiArt and the Densely Captioned Images dataset demonstrate that our pipeline delivers creative and accurate inpainting results. Our code, data, and trained models are available at https://cilabuniba.github.io/i-dream-my-painting.

Figures

Figures reproduced from arXiv: 2411.19050 by the authors.

Figure 1
Figure 1. We present a pipeline to address a novel task we refer to as [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our automatic annotation process. We in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of our pipeline. We utilize LLaVA as a prompt generator for multi-mask inpainting, integrating the generated multi [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Effects of LLaVA sampling temperature on the quality [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results. We compare the different approaches tested in our evaluation. Fine-tuning with RCA enables the model [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Automatically annotated samples from the art dataset. The first column displays the original images, along with the global [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Dataset statistics on the number of masks per example and the objects depicted in the masks (noun chunks). [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Comparative analysis on multi-mask inpainting. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 11
Figure 11. Figure 11: Visualization of our CLIPSim computation to evaluate [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 9
Figure 9. Figure 9: Qualitative comparison between repeated mask inpaint [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 12
Figure 12. Figure 12: Additional qualitative results on the Densely Captioned Images dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Additional qualitative results on the art dataset for 1-mask inpainting. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Additional qualitative results on the art dataset for 2-mask inpainting. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Additional qualitative results on the art dataset for 3-mask inpainting. [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Additional qualitative results on the art dataset for 4-mask inpainting. [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 17
Figure 17. Figure 17: Additional qualitative results on the art dataset for 5-mask inpainting. [PITH_FULL_IMAGE:figures/full_fig_p017_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 44 canonical work pages

  1. [1]

    Flamingo: A Visual Language Model for Few-Shot Learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: A Visual Language Model for Few-Shot Learning. Advances in Neural Information Processing Systems, 35:23716–23736,

    work page

  2. [2]

    Image Inpainting

    Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image Inpainting. In Proceedings of the 27th Annual Conference on Computer Graphics and Inter- active Techniques, pages 417–424, 2000. 1

    work page 2000

  3. [3]

    Im- age Multi-Inpainting via Progressive Generative Adversarial Networks

    Jiayin Cai, Changlin Li, Xin Tao, and Yu-Wing Tai. Im- age Multi-Inpainting via Progressive Generative Adversarial Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 978–987,

    work page

  4. [4]

    Leveraging Knowledge Graphs and Deep Learning for Automatic Art Analysis

    Giovanna Castellano, Vincenzo Digeno, Giovanni Sansaro, and Gennaro Vessio. Leveraging Knowledge Graphs and Deep Learning for Automatic Art Analysis. Knowledge- Based Systems, 248:108859, 2022. 5

    work page 2022

  5. [5]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, March 2023. 5

    work page 2023

  6. [6]

    Brush2Prompt: Contextual Prompt Genera- tor for Object Inpainting

    Mang Tik Chiu, Yuqian Zhou, Lingzhi Zhang, Zhe Lin, Connelly Barnes, Sohrab Amirghodsi, Eli Shechtman, and Humphrey Shi. Brush2Prompt: Contextual Prompt Genera- tor for Object Inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12636–12645, 2024. 1, 2

    work page 2024

  7. [7]

    From Posterior Sampling to Meaningful Diversity in Image Restoration

    Noa Cohen, Hila Manor, Yuval Bahat, and Tomer Michaeli. From Posterior Sampling to Meaningful Diversity in Image Restoration. In The Twelfth International Conference on Learning Representations, 2024. 2

    work page 2024

  8. [8]

    Latentpaint: Image Inpainting in Latent Space with Diffusion Models

    Ciprian Corneanu, Raghudeep Gadde, and Aleix M Mar- tinez. Latentpaint: Image Inpainting in Latent Space with Diffusion Models. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision , pages 4334–4343, 2024. 2

    work page 2024

Show all 44 references
  1. [9]

    InstructBLIP: Towards General-Purpose Vision-Language Models with Instruction Tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-Purpose Vision-Language Models with Instruction Tuning. InThirty- Seventh Conference on Neural Information Processing Sys- tems, 2023. 2

    2023

  2. [10]

    Qlora: Efficient Finetuning of Quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  3. [11]

    Diffusion Models Beat GANs on Image Synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthesis. Advances in Neural Infor- mation Processing Systems, 34:8780–8794, 2021. 2

    2021

  4. [12]

    CLIPScore: A Reference-Free Evaluation Metric for Image Captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A Reference-Free Evaluation Metric for Image Captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pages 7514–7528, 2021. 6

    2021

  5. [13]

    GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems, 30, 2017. 6

    2017

  6. [14]

    Denoising Diffu- sion Probabilistic Models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffu- sion Probabilistic Models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020. 2, 5

    2020

  7. [15]

    Classifier-Free Diffusion Guidance

    Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 5

    2021

  8. [16]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations ,

  9. [17]

    Segment Any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment Any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 5

    2023

  10. [18]

    A Diversity-Promoting Objective Function for Neural Conversation Models

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A Diversity-Promoting Objective Function for Neural Conversation Models. In Proceedings of the 2016 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Languag...

    2016

  11. [19]

    Rouge: A Package for Automatic Evaluation of Summaries

    Chin-Yew Lin. Rouge: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, 2004. 6

    2004

  12. [20]

    LLaV A-NeXT: Im- proved Reasoning, OCR, and World Knowledge, January

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A-NeXT: Im- proved Reasoning, OCR, and World Knowledge, January

  13. [21]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  14. [22]

    Pseudo Nu- merical Methods for Diffusion Models on Manifolds

    Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo Nu- merical Methods for Diffusion Models on Manifolds. In In- ternational Conference on Learning Representations , 2022. 5

    2022

  15. [23]

    Image Segmenta- tion Using Text and Image Prompts

    Timo L ¨uddecke and Alexander Ecker. Image Segmenta- tion Using Text and Image Prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7086–7096, 2022. 6, 12

    2022

  16. [24]

    Repaint: Inpainting Using Denoising Diffusion Probabilistic Models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting Using Denoising Diffusion Probabilistic Models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461–11471, 2022. 2

    2022

  17. [25]

    cGANs with Projec- tion Discriminator

    Takeru Miyato and Masanori Koyama. cGANs with Projec- tion Discriminator. In International Conference on Learning Representations, 2018. 1

    2018

  18. [26]

    GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

    Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya 9 Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In Proceedings of the 39th International Confer- e...

    2022

  19. [27]

    Bleu: A Method for Automatic Evaluation of Machine Translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311– 318, 2002. 6

    2002

  20. [28]

    Context Encoders: Feature Learning by Inpainting

    Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context Encoders: Feature Learning by Inpainting. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 2536–2544, 2016. 1

    2016

  21. [29]

    Grounding Multimodal Large Language Models to the World

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. Grounding Multimodal Large Language Models to the World. In The Twelfth International Conference on Learning Representa- tions, 2024. 2

    2024

  22. [30]

    Learning Transferable Visual Models from Natural Language Super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Super- vision. In International Conference on Machine Learning ,...

    2021

  23. [31]

    High-Resolution Image Synthesis with Latent Diffusion Models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. 1, 2, 5

    2022

  24. [32]

    LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models. Advances in Neural Infor...

  25. [33]

    Denois- ing Diffusion Implicit Models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing Diffusion Implicit Models. In International Conference on Learning Representations, 2021. 2

    2021

  26. [34]

    Resolution-Robust Large Mask Inpainting with Fourier Convolutions

    Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-Robust Large Mask Inpainting with Fourier Convolutions. In Proceedings of the IEEE/CVF Win- ...

    2022

  27. [35]

    A Picture Is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions

    Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. A Picture Is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  28. [36]

    Ex- ploring CLIP for Assessing the Look and Feel of Images

    Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Ex- ploring CLIP for Assessing the Look and Feel of Images. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pages 2555–2563, 2023. 6

    2023

  29. [37]

    Smartbrush: Text and Shape Guided Object Inpaint- ing with Diffusion Model

    Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and Shape Guided Object Inpaint- ing with Diffusion Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22428–22437, 2023. 1, 2

    2023

  30. [38]

    Freestyle Layout-to-Image Synthesis

    Han Xue, Zhiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang. Freestyle Layout-to-Image Synthesis. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14256–14266, 2023. 2, 4

    2023

  31. [39]

    Seman- tic Image Inpainting with Deep Generative Models

    Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. Seman- tic Image Inpainting with Deep Generative Models. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5485–5493, 2017. 1

    2017

  32. [40]

    Shape-Guided Object Inpainting

    Yu Zeng, Zhe Lin, and Vishal M Patel. Shape-Guided Object Inpainting. arXiv Preprint arXiv:2204.07845, 2022. 1

    2022 arXiv

  33. [41]

    Text-Guided Neural Image Inpainting

    Lisai Zhang, Qingcai Chen, Baotian Hu, and Shuoran Jiang. Text-Guided Neural Image Inpainting. In Proceedings of the 28th ACM International Conference on Multimedia , pages 1302–1310, 2020. 1

    2020

  34. [42]

    Adding Conditional Control to Text-to-Image Diffusion Models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2

    2023

  35. [43]

    The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 586–595, 2018. 6

    2018

  36. [44]

    Texygen: A Benchmarking Platform for Text Generation Models

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A Benchmarking Platform for Text Generation Models. In The 41st Interna- tional ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1097–1100, 2018. 6 10 A. A...

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.