Pith. sign in

REVIEW 4 major objections 5 minor 89 references

Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read InstaManip, an autoregressive model, claims to learn unseen image-manipulation operations from a text instruction plus a few exemplar image pairs and to apply them to new query images, outperforming prior few-shot manipulation models.

desk verdict Real architectural idea with a credible ablation, but the evaluation needs an image-disjoint test split and a CLIP-circularity check before the headline numbers can be trusted. read the letter →

arxiv 2412.01027 v2 pith:PTVC2YOB submitted 2024-12-02 cs.CV

Bolin Lai , Felix Juefei-Xu , Miao Liu , Xiaoliang Dai , Nikhil Mehta , Chenguang Zhu , Zeyi Huang , James M. Rehg
show 3 more authors
Sangmin Lee Ning Zhang Tong Xiao
This is my paper · ORCID
classification cs.CV
keywords few-shotimagemanipulationin-contextlearningautoregressivemodelgroupself-attentionvisualpromptingeditingrelationregularizationmulti-modal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

InstaManip is a multi-modal autoregressive model built for few-shot image manipulation: given a textual instruction, a query image, and one or more exemplar source-target image pairs, it learns the underlying edit operation and applies it to the query. The paper's central claim is that this in-context learning process should be split into two stages—first abstracting a manipulation representation from the text and exemplars, then applying it to the query—and that a group self-attention mechanism can enforce this split inside an autoregressive transformer. On instructions deliberately held out of training, the model reports higher alignment with both the text and the exemplars than prior few-shot manipulation models, and human raters prefer its outputs about twice as often. The paper also claims that adding more exemplars or more diverse exemplars further improves performance.

What carries the argument

The central object is the group self-attention (GSA) layer with 30 trainable manipulation tokens $Z$ inserted between the exemplar pair and the query image. The layer splits the prompt into two causally masked attention groups: the first contains the textual instruction and exemplar images and produces $Z$; the second contains $Z$, the query image, and generation tokens, which produce the output. This enforces the two-stage factorization in Eq. (2) during end-to-end training. The companion mechanism is relation regularization, which aligns the pairwise similarity of manipulation-token embeddings with the text-embedding similarity of the corresponding instructions, so that $Z$ encodes the transformation rather than irrelevant image content.

What would settle it

Construct a manipulation whose correct output depends on which object in the query is selected by the exemplar, e.g., an exemplar pair turns the left person into a pirate while a query contains two people and the manipulation should apply only to the person matching the exemplar; if the group self-attention model edits the wrong person or both, the information bottleneck at $Z$ is too narrow and the factorization fails.

Watch

Extended reading notes

Core claim

The paper introduces InstaManip, an autoregressive model that performs few-shot image manipulation through in-context learning rather than test-time optimization. It models the conditional distribution $P(Y|X,T,X',Y')$ as $P(Z|T,X',Y')\cdot P(Y|X,Z)$, where $Z$ is a set of learnable manipulation tokens that are supposed to capture the transformation independently of the query image. A group self-attention layer restricts attention so that the text and exemplar pair produce $Z$ in a first group, and the query image plus $Z$ produce the manipulated image in a second group; the exemplars and text are invisible to the generation tokens. A relation regularization loss, computed with a pre-trained text encoder, pushes manipulation embeddings of semantically similar instructions together. On a held-out set built from 30 rare-instruction keywords, the model reports text-direction alignment 19.81 and visual-example alignment 32.39 in-distribution, versus 17.13 and 27.69 for the strongest prior method, and human preference rates of 40.67 percent versus 20.00 percent. The authors' interpretation is that explicit separation of learning and applying stages improves in-context reasoning for image manipulation.

Load-bearing premise

The load-bearing premise is that a fixed set of 30 manipulation tokens can carry all the information needed to specify the edit, so that once $Z$ is known, the output is independent of the original text and exemplars.

Editorial extensions

If this is right

  • If the two-stage split is correct, autoregressive models can rival diffusion models on few-shot editing, expanding the design space for image-manipulation systems.
  • The model's improvement with more exemplars and with diverse exemplars suggests that few-shot manipulation performance is a scaling curve rather than a fixed ceiling.
  • Relation regularization provides a way to transfer textual semantic structure into visual manipulation embeddings, which could be reused for other instruction-following visual tasks.
  • Because the manipulation tokens $Z$ are the only bridge between examples and query, the same design may extend to video or multi-image manipulation by keeping the learned transformation fixed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The factorization implicitly assumes that the exemplars' role ends once $Z$ is formed; a manipulation that requires comparing the query with the exemplar at generation time would fall outside the model's inductive bias, so measured gains may come from easy-to-abstract edits rather than fine relational reasoning.
  • The text-embedding-based relation regularization couples the visual manipulation space to language semantics; for instructions whose decisive details are purely visual, this could either help or distort the representation depending on how well the text encoder captures those details.
  • The reported gains moving from two to three exemplars suggest that a dataset with more than four exemplars per instruction could reveal whether the benefit saturates or continues, an experiment the current data cannot run.
  • The same learning-and-applying decomposition could be tested as a general principle for other in-context generation problems, such as style transfer, super-resolution, or video editing, where exemplars define a mapping rather than a single task.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces InstaManip, a multi-modal autoregressive model (LLaMA-13B backbone with frozen Qwen-ViT image encoder and SDXL decoder) for few-shot image manipulation. The central idea is to decompose in-context learning into two stages: a learning stage that condenses the textual instruction and exemplar image pair into M=30 learnable manipulation tokens Z, and an applying stage that uses Z and the query image to generate the manipulated output. This is implemented via a group self-attention mechanism with separate causal attention groups, together with a relation regularization loss that aligns pairwise similarities of manipulation tokens with CLIP text-encoder similarities of the corresponding instructions. The model is trained on the InstructPix2Pix dataset with the 30 lowest-occurrence instruction keywords held out, and evaluated on the remaining 325 instructions/1296 samples, comparing against ImageBrush, VISII, PromptDiffusion, and InstructPix2Pix. Reported results show SOTA figures (in-distribution CLIP-Dir 19.81 vs 17.13 for PromptDiffusion; OOD 18.27 vs 15.41; human preference 40.67% vs 21.67% in-distribution), with ablations demonstrating contributions from both proposed components and a scaling study showing further gains from additional or more diverse exemplar pairs.

Significance. If the reported results hold, this is a meaningful advance: it is the first attempt to purpose-build an autoregressive architecture for few-shot image manipulation, and the group self-attention separation of learning and applying stages is a simple, reusable idea that could inform other in-context visual tasks. The manuscript provides a clean ablation isolating group self-attention and relation regularization, a plausible human evaluation protocol, and a qualitative analysis with failure cases. However, the evaluation has two load-bearing risks: the test set may not be image-disjoint from training (undermining the 'unseen instruction' claim), and the primary metrics CLIP-Dir/CLIP-Vis are computed in the same CLIP space used by the relation-regularization objective (part of the reported gain may be directly induced by the loss). The absence of error bars or significance tests further limits confidence in the margins.

major comments (4)
  1. [§4.1, §C.1] The test-set construction excludes instructions containing 30 selected low-occurrence keywords but does not verify that query or exemplar images are disjoint from training images. Because the InstructPix2Pix dataset reuses the same source/target images across many instructions, a test example's query or exemplar image may appear in training paired with a different, non-excluded instruction. As InstaManip is a 13B autoregressive model trained for 20,000 iterations on those images, image-level memorization could inflate the reported in-distribution CLIP-Dir, CLIP-Vis, and CLIP-T scores and the human preference rates without the model actually abstracting the transformation from the exemplar pair. Please report image-overlap statistics, evaluate on a subset with no overlapping images, or provide a memorization probe such as pairing each query with a random exemplar to show that output quality is contingent on the exemplar-derived transformation.
  2. [§3.3, Eq. (5); §4.1 metrics] The relation regularization in Eq. (5) explicitly matches the manipulation-token similarity matrix to the CLIP text-encoder similarity matrix, while the headline metrics CLIP-Dir and CLIP-Vis are computed in the same CLIP embedding space. Consequently, part of the reported improvement on these two metrics may be a direct consequence of the training objective rather than evidence of better manipulation fidelity. The paper should quantify this by reporting at least one direction/quality metric computed outside the CLIP space (e.g., a DINO-ViT-based directional score or a pixel-space direction metric) and by showing the relation-regularization ablation on such an independent metric.
  3. [Table 1, Fig. 6, §C.4] All head-to-head results are reported as point estimates without error bars, confidence intervals, or significance tests. The test set contains only 325 instructions and 1296 samples, and the user study samples 100 examples with six raters each; the claimed notable margin (e.g., CLIP-Dir 19.81 vs 17.13; human preference 40.67% vs 21.67%) could be within sampling noise. Please provide per-instruction paired bootstrap or statistical tests for the CLIP metrics and per-instance preference proportions with an appropriate test for the user study, plus inter-rater agreement.
  4. [§3.2, Eq. (2)] The factorization in Eq. (2) enforces that manipulation tokens Z are conditionally independent of the query image given the exemplars and text, and the group self-attention makes the exemplars and text invisible to generation tokens. This design discards any query-dependent reasoning about which parts of an exemplar to copy or how the transformation should interact with the query's content. The authors should state explicitly when this assumption holds in practice and show a qualitative or quantitative test that reveals its limitations, for example object replacement where the exemplar object's color or orientation should adapt to the query image's illumination and perspective.
minor comments (5)
  1. [§C.1] Typo: 'keywrods' should be 'keywords'.
  2. [Table 4 caption] Typo: 'Pleae' should be 'Please'.
  3. [§4.2] Typo: 'CLP-T' should be 'CLIP-T'.
  4. [Fig. 9] The 'out of dist.(diverse)' setting is defined only implicitly in the caption; please define it explicitly in the main text of §4.5, since the distinction between regular and diverse out-of-distribution exemplars is central to the scaling claim.
  5. [§3.1] The prompt template text should clarify whether the placeholders surround special token boundaries and whether the template tokens are encoded as text or as learnable embeddings; this would help reproducibility of the group self-attention split.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the architecture and training losses are modeling choices, not restatements of the evaluation metrics, and the main result is corroborated by an external human study.

full rationale

The paper's central claim is that splitting in-context learning into a learning stage (Z from text and exemplars) and an applying stage (Y from Z and the query) improves few-shot image manipulation. This is not derived from the evaluation metrics: Eq. (2) is presented as a modeling assumption inspired by cognitive science, and the group self-attention is an architectural implementation of that assumption. The relation regularization (Eq. 5) does use a frozen CLIP text encoder to shape the pairwise relation matrix of manipulation tokens, and the headline CLIP-Dir/CLIP-Vis metrics are computed in CLIP space; however, the regularized quantity is the self-similarity of manipulation-token embeddings against text-embedding similarities, whereas CLIP-Dir measures the cosine alignment of an image-edit direction with a caption-edit direction and CLIP-Vis measures the alignment of two image-edit directions. No equation in the paper makes these quantities equal, and the model's image encoder (Qwen ViT) is not a CLIP image encoder, so the image-side metrics are not directly optimized. The ablation in Table 2 shows that relation regularization adds only 0.85 CLIP-Dir and 1.31 CLIP-Vis over group self-attention alone, while group self-attention itself yields the larger gain; the headline claim is further supported by a human preference study (Fig. 6) that is independent of CLIP embeddings. The test set construction (Sec. 4.1 and Sec. C.1) removes all instructions containing 30 low-occurrence keywords from training, so the 'unseen instruction' condition holds by construction at the instruction level; the absence of an explicit image-disjoint check is a potential data-leakage/correctness concern, not a circularity. Self-citations (e.g., [13], [29], [65]) appear only in related-work context and are not load-bearing for the reported results. No circular step satisfying the 'quote and exhibit the reduction' standard was found.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claim rests on the validity of the learning/applying factorization, the CLIP-based relation regularization, and a test set built by keyword filtering plus manual cleanup. The main free parameters are the number of manipulation tokens (chosen by test-set ablation) and the regularization weight.

free parameters (2)
  • Number of manipulation tokens M = 30
    Selected based on test-set ablation (Table 4): CLIP-Dir and CLIP-Vis peak at 30 tokens; the central results depend on this choice.
  • Relation regularization weight alpha = 0.1
    Set in Sec C.2 without a sensitivity ablation; balances reconstruction loss and CLIP-similarity regularization.
assumptions (5)
  • domain assumption The factorization P(Y|X,T,X',Y') = P(Z|T,X',Y') * P(Y|X,Z) is valid for image manipulation.
    Used to justify group self-attention (Eq. 2, Sec 3). If the true conditional requires joint reasoning over query and exemplar, the split discards useful information.
  • domain assumption CLIP text embedding similarity is a good proxy for manipulation-operation similarity.
    Relation regularization (Eq. 5) enforces manipulation-token relations to match CLIP text relations.
  • domain assumption Filtering training data by removing all instructions containing 30 low-occurrence keywords creates a valid unseen-instruction test split.
    Test set construction (Sec 4.1, C.1); assumes no leakage via paraphrases or multi-word variants.
  • domain assumption The InstructPix2Pix dataset ground truth is correct enough after manual filtering.
    Manual removal of 'incorrect ground truth' (Sec C.1) is a subjective quality filter applied before evaluation.
  • domain assumption Frozen Qwen ViT image encoder and SDXL decoder are sufficient for the manipulation task.
    These components are frozen during training and bound the information capacity of the pipeline.
invented entities (1)
  • Manipulation tokens Z
    purpose: Learnable token embeddings that abstract the manipulation operation from text and exemplar pairs, serving as the only bridge between the learning and applying attention groups.
    They are an internal latent representation; their existence is only observable through downstream generation performance, with no external falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation." pith.science (2026). https://pith.science/paper/PTVC2YOB

@misc{pith2026241201027,
  author       = {Pith},
  title        = {Pith review of: Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PTVC2YOB}},
  note         = {Machine review of arXiv:2412.01027}
}
abstract

Text-guided image manipulation has experienced notable advancement in recent years. In order to mitigate linguistic ambiguity, few-shot learning with visual examples has been applied for instructions that are underrepresented in the training set, or difficult to describe purely in language. However, learning from visual prompts requires strong reasoning capability, which diffusion models are struggling with. To address this issue, we introduce a novel multi-modal autoregressive model, dubbed $\textbf{InstaManip}$, that can $\textbf{insta}$ntly learn a new image $\textbf{manip}$ulation operation from textual and visual guidance via in-context learning, and apply it to new query images. Specifically, we propose an innovative group self-attention mechanism to break down the in-context learning process into two separate stages -- learning and applying, which simplifies the complex problem into two easier tasks. We also introduce a relation regularization method to further disentangle image transformation features from irrelevant contents in exemplar images. Extensive experiments suggest that our method surpasses previous few-shot image manipulation models by a notable margin ($\geq$19% in human evaluation). We also find our model can be further boosted by increasing the number or diversity of exemplar images.

Figures

Figures reproduced from arXiv: 2412.01027 by the authors.

Figure 1
Figure 1. When learning a new image manipulation operation that is [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of InstructPix2Pix [7] and our model. We exclude “Lamborghini” from training set for both models. complexity, which leads to a bottleneck in learning the de￾sired manipulation rules and transferring to other images. To address these issues in few-shot image manipulation, we introduce InstaManip, an innovative multi-modal au￾toregressive architecture that models the two stages sepa￾rately. Specifically, we… view at source ↗
Figure 3
Figure 3. Comparison of the performance of plain self-attention [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Overview of the proposed InstaManip architecture (left) and group self-attention mechanism (right, represented by query-key [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison with InstructPix2Pix and previous few-shot image manipulation methods. All instructions containing [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Human evaluation (represented in preference rate) of our [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Examples of InstaManip output. Our model learns trans [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 10
Figure 10. Figure 10: Demonstration of different manipulation on the same [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 9
Figure 9. Figure 9: The performance of our model with different numbers [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 11
Figure 11. Figure 11: Comparison of the four CLIP-based metrics on the out [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: The visualization of manipulating the query image using the same textual instruction, yet different visual examples. When [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Additional visualization of the output from InstaManip. All instructions containing selected keywords (highlighted in [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: More demonstration of the output from InstaManip (continuation of Fig. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: The interface used for human evaluation. The four manipulated images are randomly shuffled to avoid potential bias. Please [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Failure cases of InstaManip. Please refer to Sec. [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

89 extracted references · 47 canonical work pages

  1. [1]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 5

    arXiv 2023

  2. [2]

    Sequential modeling enables scalable learn- ing for large vision models

    Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learn- ing for large vision models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22861–22872, 2024. 2, 3, 4

    2024

  3. [3]

    ediff-i: Text-to-image dif- fusion models with an ensemble of expert denoisers

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Ji- aming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image dif- fusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022. 2

    arXiv 2022

  4. [4]

    Towards in-context scene understanding

    Ivana Bala ˇzevi´c, David Steiner, Nikhil Parthasarathy, Relja Arandjelovi´c, and Olivier J H ´enaff. Towards in-context scene understanding. In Proceedings of the 37th Interna- tional Conference on Neural Information Processing Sys- tems, pages 63758–63778, 2023. 3

    2023

  5. [5]

    Visual prompting via image inpaint- ing

    Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Glober- son, and Alexei A Efros. Visual prompting via image inpaint- ing. In Proceedings of the 36th International Conference on Neural Information Processing Systems , pages 25005– 25017, 2022. 3

    2022

  6. [6]

    Ledits++: Limitless image editing using text-to-image models

    Manuel Brack, Felix Friedrich, Katharia Kornmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolin´ario Passos. Ledits++: Limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 8861–8870, 2024. 2

    2024

  7. [7]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 2, 3, 5, 6, 13, 17, 18

    2023

  8. [8]

    Lan- guage models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Pro- cessing Systems, pages 1877–1901, 2020. 3, 8

    1901

Show all 89 references
  1. [9]

    Enhancing diffu- sion models with text-encoder reinforcement learning

    Chaofeng Chen, Annan Wang, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. Enhancing diffu- sion models with text-encoder reinforcement learning. In European Conference on Computer Vision, pages 182–198. Springer, 2024. 2

    2024

  2. [10]

    Gentron: Diffusion trans- formers for image and video generation

    Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Diffusion trans- formers for image and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  3. [11]

    Scaling instruction- finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. 2

    2024

  4. [12]

    Diffedit: Diffusion-based semantic image editing with mask guidance

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. In The Eleventh International Conference on Learning Representations, 2023. 2

    2023

  5. [13]

    Emu: Enhanc- ing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xi- aofang Wang, Abhimanyu Dubey, et al. Emu: Enhanc- ing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023. 2

    2023 arXiv

  6. [14]

    Dreamllm: Synergistic multimodal com- prehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal com- prehension and creation. In The Twelfth International Con- ference on Learning Representations, 2024. 3

    2024

  7. [15]

    Diffusion self-guidance for control- lable image generation

    Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffusion self-guidance for control- lable image generation. In Proceedings of the 37th Inter- national Conference on Neural Information Processing Sys- tems, pages 16222–16239, 2023. 2

    2023

  8. [16]

    Fluid: Scaling autoregressive text-to-image generative models with continuous tokens

    Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.13863, 2024. 3

    arXiv

  9. [17]

    Puma: Empowering unified mllm with multi-granular visual generation

    Rongyao Fang, Chengqi Duan, Kun Wang, Hao Li, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, Hongsheng Li, and Xihui Liu. Puma: Empowering unified mllm with multi-granular visual generation. arXiv preprint arXiv:2410.13861, 2024. 2, 3

    arXiv

  10. [18]

    Explore in-context learning for 3d point cloud understanding

    Zhongbin Fang, Xiangtai Li, Xia Li, Joachim M Buhmann, Chen Change Loy, and Mengyuan Liu. Explore in-context learning for 3d point cloud understanding. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 42382–42395, 2023. 3

    2023

  11. [19]

    Making llama see and draw with seed tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. InThe Twelfth International Conference on Learning Representations, 2024. 3

    2024

  12. [20]

    Seed-x: Mul- 9 timodal models with unified multi-granularity comprehen- sion and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Mul- 9 timodal models with unified multi-granularity comprehen- sion and generation. arXiv preprint arXiv:2404.14396, 2024. 2, 3, 4

    2024 arXiv

  13. [21]

    Analogist: Out-of-the-box visual in-context learning with image diffusion model

    Zheng Gu, Shiyuan Yang, Jing Liao, Jing Huo, and Yang Gao. Analogist: Out-of-the-box visual in-context learning with image diffusion model. ACM Transactions on Graphics (TOG), 43(4):1–15, 2024. 3

    2024

  14. [22]

    Generative visual instruction tuning

    Jefferson Hernandez, Ruben Villegas, and Vicente Or- donez. Generative visual instruction tuning. arXiv preprint arXiv:2406.11262, 2024. 2, 3

    2024 arXiv

  15. [23]

    Prompt-to-prompt image editing with cross-attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. In The Eleventh Inter- national Conference on Learning Representations, 2023. 2

    2023

  16. [24]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. In International Conference on Learning Representations, 2022. 5

    2022

  17. [25]

    Multimodal task vectors enable many-shot multimodal in-context learn- ing

    Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, and Roei Herzig. Multimodal task vectors enable many-shot multimodal in-context learn- ing. arXiv preprint arXiv:2406.15334, 2024. 2, 3

    2024 arXiv

  18. [26]

    Customizing text-to-image models with a single image pair

    Maxwell Jones, Sheng-Yu Wang, Nupur Kumari, David Bau, and Jun-Yan Zhu. Customizing text-to-image models with a single image pair. arXiv preprint arXiv:2405.01536, 2024. 3

    2024 arXiv

  19. [27]

    Chameleon: A data-efficient gener- alist for dense visual prediction in the wild

    Donggyun Kim, Seongwoong Cho, Semin Kim, Chong Luo, and Seunghoon Hong. Chameleon: A data-efficient gener- alist for dense visual prediction in the wild. In European Conference of Computer Vision, 2024. 3

    2024

  20. [28]

    Gen- erating images with multimodal language models

    Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Gen- erating images with multimodal language models. In Pro- ceedings of the 37th International Conference on Neural In- formation Processing Systems, pages 21487–21506, 2023. 3

    2023

  21. [29]

    Lego: Learning egocentric action frame generation via visual instruction tuning

    Bolin Lai, Xiaoliang Dai, Lawrence Chen, Guan Pang, James M Rehg, and Miao Liu. Lego: Learning egocentric action frame generation via visual instruction tuning. In Eu- ropean Conference of Computer Vision, 2024. 2

    2024

  22. [30]

    Blip-diffusion: pre-trained subject representation for controllable text-to- image generation and editing

    Dongxu Li, Junnan Li, and Steven CH Hoi. Blip-diffusion: pre-trained subject representation for controllable text-to- image generation and editing. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 30146–30166, 2023. 2

    2023

  23. [31]

    Visual in-context prompting

    Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Jianwei Yang, Chunyuan Li, et al. Visual in-context prompting. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12861–12871, 2024. 3

    2024

  24. [32]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. In Proceedings of the 38th Interna- tional Conference on Neural Information Processing Sys- tems, 2024. 3, 4

    2024

  25. [33]

    Visual atribute transfer through deep image analogy

    Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. Visual atribute transfer through deep image analogy. ACM Transactions on Graphics, 36(4):120, 2017. 2

    2017

  26. [34]

    Text-driven image editing via learn- able regions

    Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Lu Jiang, and Ming-Hsuan Yang. Text-driven image editing via learn- able regions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7059– 7068, 2024. 2

    2024

  27. [35]

    Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining

    Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408.02657, 2024. 3

    2024 arXiv

  28. [36]

    Glid: Pre-training a generalist encoder-decoder vision model

    Jihao Liu, Jinliang Zheng, Yu Liu, and Hongsheng Li. Glid: Pre-training a generalist encoder-decoder vision model. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 22851–22860, 2024. 3

    2024

  29. [37]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2017. 18

    2017

  30. [38]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceed- ings of the IEEE/CVF Conference on Computer Vision ...

    2024

  31. [39]

    Star: Scale-wise text-to- image generation via auto-regressive representations

    Xiaoxiao Ma, Mohan Zhou, Tao Liang, Yalong Bai, Tiejun Zhao, Huaian Chen, and Yi Jin. Star: Scale-wise text-to- image generation via auto-regressive representations. arXiv preprint arXiv:2406.10797, 2024. 3

    2024 arXiv

  32. [40]

    Sdedit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equa- tions. In International Conference on Learning Representa- tions, 2022. 2

    2022

  33. [41]

    Watch your steps: Local image and scene editing by text instructions

    Ashkan Mirzaei, Tristan Aumentado-Armstrong, Marcus A Brubaker, Jonathan Kelly, Alex Levinshtein, Konstantinos G Derpanis, and Igor Gilitschenski. Watch your steps: Local image and scene editing by text instructions. In European Conference on Computer Vision , pages 111–129. S...

    2024

  34. [42]

    Null-text inversion for editing real im- ages using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real im- ages using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023. 2

    2023

  35. [43]

    Visual instruction inversion: image editing via visual prompting

    Thao Nguyen, Yuheng Li, Utkarsh Ojha, and Yong Jae Lee. Visual instruction inversion: image editing via visual prompting. In Proceedings of the 37th International Con- ference on Neural Information Processing Systems , pages 9598–9613, 2023. 2, 3, 5, 6, 8, 18

    2023

  36. [44]

    Swiftbrush: One-step text-to-image diffusion model with variational score distilla- tion

    Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step text-to-image diffusion model with variational score distilla- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 7807–7816,

  37. [45]

    In- context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In- context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022. 2

    2022 arXiv

  38. [46]

    Editing implicit assumptions in text-to-image diffusion models

    Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to-image diffusion models. In 10 Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7053–7061, 2023. 2

    2023

  39. [47]

    Effective real image editing with accelerated iter- ative diffusion inversion

    Zhihong Pan, Riccardo Gherardi, Xiufeng Xie, and Stephen Huang. Effective real image editing with accelerated iter- ative diffusion inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15912– 15921, 2023. 2

    2023

  40. [48]

    Precisecontrol: En- hancing text-to-image diffusion models with fine-grained at- tribute control

    Rishubh Parihar, VS Sachidanand, Sabariswaran Mani, Te- jan Karmali, and R Venkatesh Babu. Precisecontrol: En- hancing text-to-image diffusion models with fine-grained at- tribute control. InEuropean Conference on Computer Vision, pages 469–487. Springer, 2025. 2

    2025

  41. [49]

    True few- shot learning with language models

    Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few- shot learning with language models. In Proceedings of the 35th International Conference on Neural Information Pro- cessing Systems, pages 11054–11070, 2021. 8

    2021

  42. [50]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth Interna- tional Conference on Learning Representations, 2024. 5

    2024

  43. [51]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  44. [52]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 2

    2021

  45. [53]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022

    2022 arXiv

  46. [54]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2

    2022

  47. [55]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  48. [56]

    Towards more unified in-context visual un- derstanding

    Dianmo Sheng, Dongdong Chen, Zhentao Tan, Qiankun Liu, Qi Chu, Jianmin Bao, Tao Gong, Bin Liu, Shengwei Xu, and Nenghai Yu. Towards more unified in-context visual un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13362...

    2024

  49. [57]

    Emu edit: Precise image editing via recognition and gen- eration tasks

    Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and gen- eration tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pa...

    2024

  50. [58]

    Diffusion image analo- gies

    Ad ´ela ˇSubrtov´a, Michal Luk ´aˇc, Jan ˇCech, David Futschik, Eli Shechtman, and Daniel S `ykora. Diffusion image analo- gies. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1–10, 2023. 2

    2023

  51. [59]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 3

    2024 arXiv

  52. [60]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, 2023. 3, 4

    2023

  53. [61]

    Generative multimodal mod- els are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal mod- els are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2024

  54. [62]

    Imagebrush: learning visual in-context instructions for exemplar-based image manipulation

    Yasheng Sun, Yifan Yang, Houwen Peng, Yifei Shen, Yuqing Yang, Han Hu, Lili Qiu, and Hideki Koike. Imagebrush: learning visual in-context instructions for exemplar-based image manipulation. In Proceedings of the 37th Interna- tional Conference on Neural Information Processing ...

    2023

  55. [63]

    Rethinking and improving visual prompt selection for in-context learning segmentation

    Wei Suo, Lanqing Lai, Mengyang Sun, Hanwang Zhang, Peng Wang, and Yanning Zhang. Rethinking and improving visual prompt selection for in-context learning segmentation. In European Conference on Computer Vision, pages 18–35. Springer, 2024. 3

    2024

  56. [64]

    Cognitive load during problem solving: Ef- fects on learning

    John Sweller. Cognitive load during problem solving: Ef- fects on learning. Cognitive science, 12(2):257–285, 1988. 2, 3

    1988

  57. [65]

    Codi-2: In-context inter- leaved and interactive any-to-any generation

    Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. Codi-2: In-context inter- leaved and interactive any-to-any generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 27425–27434, 2024. 2, 3

    2024

  58. [66]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 3

    2024 arXiv

  59. [67]

    How to grow a mind: Statistics, structure, and abstraction

    Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow a mind: Statistics, structure, and abstraction. science, 331(6022):1279–1285,

  60. [68]

    Visual autoregressive modeling: Scalable image gen- eration via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image gen- eration via next-scale prediction. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024. 3

    2024

  61. [69]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 5

    2023 arXiv

  62. [70]

    Edict: Exact diffusion inversion via coupled transformations

    Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. In Proceed- 11 ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22532–22541, 2023. 2

    2023

  63. [71]

    Explore in-context segmentation via latent diffusion models

    Chaoyang Wang, Xiangtai Li, Henghui Ding, Lu Qi, Jiangn- ing Zhang, Yunhai Tong, Chen Change Loy, and Shuicheng Yan. Explore in-context segmentation via latent diffusion models. arXiv preprint arXiv:2403.09616, 2024. 3

    2024 arXiv

  64. [72]

    Images speak in images: A generalist painter for in-context visual learning

    Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6830–6839, 2023. 3

    2023

  65. [73]

    Seggpt: Segmenting ev- erything in context

    Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting ev- erything in context. InInternational Conference of Computer Vision, 2023. 3

    2023

  66. [74]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 2, 3

    2024 arXiv

  67. [75]

    In-context learning unlocked for diffu- sion models

    Zhendong Wang, Yifan Jiang, Yadong Lu, Yelong Shen, Pengcheng He, Weizhu Chen, Zhangyang Wang, and Mingyuan Zhou. In-context learning unlocked for diffu- sion models. In Proceedings of the 37th International Con- ference on Neural Information Processing Systems , pages 8542–85...

    2023

  68. [76]

    The learn- ability of in-context learning

    Noam Wies, Yoav Levine, and Amnon Shashua. The learn- ability of in-context learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 36637–36651, 2023. 2

    2023

  69. [77]

    Omnigen: Unified image generation

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xin- grun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv preprint arXiv:2409.11340, 2024. 3

    2024 arXiv

  70. [78]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. 3

    2024 arXiv

  71. [79]

    To- wards global optimal visual in-context learning prompt se- lection

    Chengming Xu, Chen Liu, Yikai Wang, and Yanwei Fu. To- wards global optimal visual in-context learning prompt se- lection. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024. 3

    2024

  72. [80]

    Improv: Inpainting-based multimodal prompting for computer vision tasks

    Jiarui Xu, Yossi Gandelsman, Amir Bar, Jianwei Yang, Jian- feng Gao, Trevor Darrell, and Xiaolong Wang. Improv: Inpainting-based multimodal prompting for computer vision tasks. Transactions on Machine Learning Research, 2024. 3

    2024

  73. [81]

    Prompt-free diffusion: Taking” text” out of text-to-image diffusion models

    Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Ir- fan Essa, and Humphrey Shi. Prompt-free diffusion: Taking” text” out of text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 8682–8692, 2024. 3

    2024

  74. [82]

    Anygpt: Unified multimodal llm with dis- crete sequence modeling

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal llm with dis- crete sequence modeling. arXiv preprint arXiv:2402.12226,

    arXiv

  75. [83]

    Magicbrush: a manually annotated dataset for instruction- guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: a manually annotated dataset for instruction- guided image editing. In Proceedings of the 37th Interna- tional Conference on Neural Information Processing Sys- tems, pages 31428–31449, 2023. 2

    2023

  76. [84]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2, 3

    2023

  77. [85]

    What makes good examples for visual in-context learning? In Proceed- ings of the 37th International Conference on Neural Infor- mation Processing Systems, pages 17773–17794, 2023

    Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? In Proceed- ings of the 37th International Conference on Neural Infor- mation Processing Systems, pages 17773–17794, 2023. 3

    2023

  78. [86]

    Instructbrush: Learning attention-based in- struction optimization for image editing

    Ruoyu Zhao, Qingnan Fan, Fei Kou, Shuai Qin, Hong Gu, Wei Wu, Pengcheng Xu, Mingrui Zhu, Nannan Wang, and Xinbo Gao. Instructbrush: Learning attention-based in- struction optimization for image editing. arXiv preprint arXiv:2403.18660, 2024. 2, 3

    2024 arXiv

  79. [87]

    Calibrate before use: Improving few-shot perfor- mance of language models

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot perfor- mance of language models. In International conference on machine learning, pages 12697–12706. PMLR, 2021. 2

    2021

  80. [88]

    Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. 3 12 Unlea...

    2024 arXiv

  81. [89]

    As a baseline of text-guided image editing model, InstructPix2Pix is trained only with textual instructions

    and PromptDiffusion [75]. As a baseline of text-guided image editing model, InstructPix2Pix is trained only with textual instructions. The model weights are also used for VISII, which relies on a pre-trained InstructPix2Pix model for test-time finetuning. We freeze the weights...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.