Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

AR-RAG: Autoregressive Retrieval Augmentation for Image Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that retrieving and blending image patches at every autoregressive decoding step improves generated-image fidelity and prompt adherence, with its best configuration reaching 6.67 FID on Midjourney-30K and 0.78 on GenEval.

desk verdict Patch-level autoregressive retrieval is a real idea and DAiD gives clean training-free evidence, but the headline FAiD numbers are confounded by domain-matched fine-tuning and lack a no-retrieval control. read the letter →

arxiv 2506.06962 v3 pith:QEKHC4LC submitted 2025-06-08 cs.CV

Jingyuan Qi , Zhiyang Xu , Qifan Wang , Lifu Huang This is my paper
classification cs.CV
keywords retrieval-augmentedgenerationautoregressiveimagepatch-levelretrievaltext-to-imagedecoding-timeaugmentationk-nearestneighbordiscretetokensvisualfidelity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that retrieval-augmented image generation has been retrieving at the wrong granularity and the wrong time. Rather than fetching whole reference images once before decoding, AR-RAG queries a database of image patches at every step, using the patches the model has already drawn as the search key and blending the retrieved patch-level references into the next-token prediction. On standard benchmarks this fixes the failure modes of static image-level retrieval (over-copying irrelevant details, stylistic bias, and missed prompt elements) while improving fidelity and instruction following: Janus-Pro's FID on Midjourney-30K drops from 14.33 to 9.15 with the training-free DAiD and to 6.67 with the fine-tuned FAiD, and the GenEval overall score rises from 0.71 to 0.78. The same machinery also improves a masked-token backbone, Show-o, indicating the benefit is not tied to strict left-to-right decoding.

What carries the argument

The load-bearing object is a patch-level retrieval database built from the same quantized autoencoder the generator uses: real images are encoded into grids of latent patch vectors, and each database entry stores one patch vector as its value together with the concatenated vectors of its h-hop surrounding patches as its key. At decoding time the query is built the same way from the patches the model has already generated, the top K entries are selected by L2 distance, and their stored patch vectors are mapped back to codebook tokens. In DAiD those tokens re-weight the next-token distribution; in FAiD they are embedded, smoothed by multi-scale convolutions against the partial image, and blended into the hidden state with learned compatibility scores. This machinery converts the generator's own latent geometry into a reusable, non-parametric memory of real-image patch statistics.

What would settle it

Two concrete checks would settle the mechanism. First, in DAiD, replace the top-K retrieved patches with K patches drawn uniformly at random from the codebook while keeping everything else fixed: if FID on Midjourney-30K stays near 9.15, then the retrieval signal itself is not what drives the gain. Second, measure retrieval precision during real generation: at typical decoding steps, build the h-hop query from partially generated images and compare the L2 distance of the top-K retrieved patch vectors to the eventual ground-truth patch against a baseline of randomly sampled tokens; if the retrieved set is no closer than random once generation is under way, the context-aware retrieval claimed by AR-RAG is not reaching the decoder.

Watch

Extended reading notes

Core claim

The paper's central claim is that context-aware, patch-level retrieval performed during decoding improves autoregressive image generation, and that two concrete mechanisms realize this: a training-free decoding strategy (DAiD, Distribution-Augmentation in Decoding) and a parameter-efficient fine-tuning strategy (FAiD, Feature-Augmentation in Decoding). In DAiD, the top-K database patches whose h-hop neighborhoods match the already-generated neighborhood of the next token are converted into a sparse probability distribution over codebook tokens, weighted by inverse L2 distance with a temperature, and merged with the model's own next-token distribution. In FAiD, the retrieved patch embeddings are mapped into the model's hidden space, refined by multi-scale convolutions against the partial image, scored for compatibility with the local context, and added into the next token's hidden state at several decoder layers. The experimental claim is that both variants beat their base models on fidelity and alignment metrics, that FAiD on Janus-Pro sets a new state of the art among comparable-scale autoregressive models (6.67 FID on Midjourney-30K, 0.78 overall on GenEval), and that previous image-level retrieval methods (RDM, ImageRAG, a replicated RA-CM3) often underperform their non-retrieval counterparts precisely because a single static retrieval over-constrains generation.

Load-bearing premise

The method assumes that image patches whose h-hop surrounding contexts are close in the tokenizer's latent space have contents worth copying for the next patch, and that this stays true when the query is assembled from the model's own partially generated image, which can drift from the statistics of the real photographs the database was built from.

Editorial extensions

If this is right

  • If the central claim holds, a training-free decoding change (DAiD) recovers a large share of the fidelity gain at roughly 0.22 percent added inference time, making patch-level retrieval a practical drop-in option for deployed autoregressive generators.
  • The reported gains are largest where static image retrieval fails hardest: multi-object prompts, spatial relations, rare object combinations, and detailed prompts.
  • Because the same retrieval database and blending modules also improve Show-o, a masked-token model, the mechanism transfers across discrete-token architectures and does not require strict left-to-right decoding.
  • At comparable parameter counts, the paper's numbers place FAiD ahead of diffusion and autoregressive baselines several times its size on GenEval and DPG-Bench.
  • Patch-level retrieval preserves compositional flexibility: relevant objects are incorporated while context from reference images that would normally be copied, such as branches, logos, and fixed layouts, is left out.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The optimized retrieval weight for DAiD is small (lambda = 0.05), which I read as evidence that the active ingredient is a mild sharpening of the model's codebook distribution toward real-image patch statistics rather than substitution of retrieved content; if so, DAiD's benefit should grow with codebook coverage and shrink as the base model's own distribution sharpens, both of which are testable
  • The paper validates retrieval only against ground-truth tokens from real images, so a natural follow-up is to measure retrieval precision when queries are built from the model's partial generations, where distribution drift could turn retrieved patches into error reinforcement.
  • Because the database is built with the frozen image tokenizer and depends only on latent neighborhood geometry, the same index could serve any discrete-token generator, and queries could alternatively be formed from the language model's hidden states rather than from codebook vectors; the paper builds queries only from codebook vectors.
  • The database covers roughly 13.6 million images and the paper identifies scale as a limitation; if the mechanism's value is supplying rare-pattern statistics, gains should scale with database diversity, an experiment the authors did not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AR-RAG, a patch-level autoregressive retrieval augmentation paradigm for image generation. It introduces two instantiations: DAiD, a training-free decoding strategy that merges the model's next-patch distribution with a distribution over retrieved patch tokens, and FAiD, a parameter-efficient fine-tuning approach that refines retrieved patch features with multi-scale convolutions and blends them into the hidden states of a transformer-based autoregressive model. The methods are evaluated on GenEval, DPG-Bench, and Midjourney-30K, with Janus-Pro and Show-o as backbones. The headline claims are that Janus-Pro+FAiD achieves 6.67 FID on Midjourney-30K and 0.78 overall GenEval score, 'a new state of the art among autoregressive image generation models of comparable scale,' and that DAiD provides consistent training-free gains (e.g., FID 14.33 to 9.15 on Janus-Pro).

Significance. If the results hold, the paper makes a useful conceptual contribution by moving retrieval augmentation from static, image-level conditioning to dynamic, patch-level, autoregressive retrieval. DAiD is particularly attractive because it is training-free, plug-and-play, and shows consistent gains across two backbones and three benchmarks, providing independent evidence that the retrieval mechanism itself helps. FAiD, if its gains could be cleanly attributed to retrieval rather than to fine-tuning on distributionally close data, would be a strong parameter-efficient method with practical appeal. However, as presented, the FAiD results are confounded by a 50K-pair fine-tuning recipe that includes Midjourney-v6 images, which are distributionally close to the Midjourney-30K evaluation set, and no control without retrieval is reported. The significance is therefore partial: the retrieval paradigm is promising and the DAiD evidence is solid, but the SOTA claim rests on the confounded FAiD numbers.

major comments (3)
  1. [§4, Appendix B.3, Tables 1–3] The FAiD results are confounded with fine-tuning on the evaluation distribution. FAiD fine-tunes Janus-Pro on 50K image-caption pairs sampled from CC12M and Midjourney-v6 (25K from each), while the headline Midjourney-30K benchmark is drawn from MJHQ-30K, a dataset of Midjourney images. Under these conditions, the reported FID improvement from 14.33 to 6.67 and the GenEval improvement from 0.71 to 0.78 cannot be attributed specifically to retrieval augmentation: fine-tuning on a distributionally close corpus could produce much of this gain even without any retrieval. The paper does not report the necessary control, namely Janus-Pro fine-tuned on the same 50K pairs with the same schedule and hyperparameters but with FAiD modules removed or retrieved-patch contributions zeroed. Without this control, the central claim that FAiD's retrieval mechanism drives the SOTA result is not established.
  2. [§3.2, §3.3, Appendix C.1] The retrieval assumption is only validated against ground-truth training tokens, not against the actual generation setting. Appendix C.1 (Figure 6) shows that top-K retrieved tokens are closer in l2 distance to ground-truth tokens than randomly sampled tokens, which supports the basic neighborhood assumption. However, during generation the retrieval queries are built from partially generated images, which can drift from real-image statistics, and the paper does not measure whether retrieval quality degrades in this setting. Furthermore, the blending step in Eq. (7) is never isolated: the paper does not test whether adding the retrieved-feature term improves generation relative to the same fine-tuned model without that term. A concrete test would be to report retrieval accuracy (or generation FID/CMMD) when queries are taken from generated contexts instead of real-image contexts, or to ablate retrieval at early versus late generation steps.
  3. [Appendix C.2] The main hyperparameters are selected by optimizing FID on the Midjourney-10K subset, and the reported Midjourney-30K results are on the same benchmark family. Specifically, λ and τ for DAiD and hop and blender count b for FAiD are chosen from Figure 7 based on FID on Midjourney-10K. Because Midjourney-10K is a subset of the same MJHQ distribution as the 30K evaluation set, this constitutes model selection on a near-distribution validation set. The paper should either report results with a fixed default configuration or demonstrate that the chosen hyperparameters transfer across benchmarks without dataset-specific selection. Otherwise, part of the reported gains, especially on Midjourney-30K, may reflect tuning to the evaluation benchmark's distribution rather than a general property of the retrieval method.
minor comments (5)
  1. [Eq. (2)] The condition 'if v = ˆvk for some m ∈ {1,2,...,K}' uses m where it should use k; the index in the set should match the index in the retrieved token, e.g., 'for some k ∈ {1,2,...,K}'.
  2. [§3.3, Eq. (7)] The sentence 'The final representation for the next image token vij after layer j' should refer to layer l (or l+1), since j already denotes the patch column index.
  3. [§6] There is a typo: 'All of these works differe from our method' should be 'differ from our method'.
  4. [§2, §B.2] The main text says DAiD and FAiD can be extended to models 'such as LlamaGen, Show-o and VAR,' but Show-o is a masked-token model rather than a strictly left-to-right autoregressive model. Appendix B.2 correctly describes the necessary adaptations; the main text should acknowledge this distinction to avoid overstating generality.
  5. [Appendix A, Algorithm 1] The algorithm uses Conv1_{q×q} and Conv2_{q×q} but does not connect them to the learnable scale weights Ω in Eq. (6). It should clarify which parameters are trained and how the two-stage convolutions relate to the softmax-weighted combination in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: AR-RAG's headline numbers are empirical measurements against external benchmarks; the FAiD fine-tuning confound is a validity concern, not a derivation-level circularity.

full rationale

The paper's central claim—patch-level autoregressive retrieval improves image generation—is established by direct comparison of DAiD/FAiD against unmodified Janus-Pro and Show-o backbones on external benchmarks (GenEval, DPG-Bench, Midjourney-30K). DAiD is a training-free decoding modification; its gains (e.g., FID 14.33→9.15 on Midjourney-30K for Janus-Pro) are measured, not derived from the retrieval construction. FAiD's inserted modules are trained with the same autoregressive objective as Eq. (1), and retrieval signals enter as additional context; no equation in the paper defines the reported FID/CMMD/GenEval scores as algebraically equivalent to the retrieval objective or to the hyperparameter choices. Hyperparameters (λ, τ, h, b) are tuned on a Midjourney-10K subset and then fixed for the reported benchmarks, which is standard model selection rather than a fitted-input-called-prediction reduction. The k-NN assumption in Appendix C.1 is explicitly stated as an assumption and empirically sanity-checked, not imported from a self-citation. The cited prior work by the authors appears only as related work and is not load-bearing. The main substantive weakness is experimental: FAiD is fine-tuned on 50K pairs including 25K Midjourney-v6 images without a retrieval-disabled, same-recipe control, so the headline FAiD gain cannot be cleanly attributed to retrieval; this is an experimental control/confound concern, not a circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of codebook-space patch retrieval from real images and on hyperparameters selected on the benchmark family. No new physical entities are introduced.

free parameters (6)
  • lambda (retrieval weight for DAiD) = 0.05
    Chosen on Midjourney-10K FID in Appendix C.2; controls how much the retrieved patch distribution affects the final token distribution.
  • tau (retrieval temperature for DAiD) = 0.6
    Chosen on Midjourney-10K FID in Appendix C.2; controls sharpness of the softmax over retrieval distances.
  • h (retrieval neighborhood hop level) = composite '12'
    Chosen on Midjourney-10K FID in Appendix C.2; determines how many surrounding patch contexts form the retrieval key.
  • b (number of FAiD blender modules) = 2
    Chosen on Midjourney-10K FID in Appendix C.2; sets how many decoder layers receive FAiD modules.
  • K (number of retrieved patches) = not reported
    Top-K is a central design choice in both DAiD and FAiD, but the paper never states the K used in experiments.
  • Q (maximum multi-scale convolution kernel size) = not reported
    The multi-scale smoothing in FAiD goes from 2x2 to QxQ, but Q is never specified.
assumptions (4)
  • domain assumption Image patches with similar surrounding latent contexts have similar center content.
    Explicitly stated as an assumption in Appendix C.1 and used to justify the whole retrieval database design in Section 3.1.
  • domain assumption The h-hop concatenated patch vector is a sufficient representation of local context for retrieval.
    Section 3.1 defines database keys as h-hop surrounding patch vectors; no alternative context representation is tested.
  • domain assumption The partially generated image context, including zero placeholders, can serve as a reliable retrieval query.
    Sections 3.2 and 3.3 build queries from generated patches and zero vectors; the paper itself notes DAiD suffers from noise propagation.
  • domain assumption CC12M, JourneyDB, and DataComp contain patch-level visual references that transfer to the evaluation prompts.
    Section 4 builds the retrieval database from these sources; no analysis of coverage or domain mismatch beyond exclusion of test samples is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AR-RAG: Autoregressive Retrieval Augmentation for Image Generation." pith.science (2026). https://pith.science/paper/QEKHC4LC

@misc{pith2026250606962,
  author       = {Pith},
  title        = {Pith review of: AR-RAG: Autoregressive Retrieval Augmentation for Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QEKHC4LC}},
  note         = {Machine review of arXiv:2506.06962}
}
read the original abstract

We introduce Autoregressive Retrieval Augmentation (AR-RAG), a novel paradigm that enhances image generation by autoregressively incorporating knearest neighbor retrievals at the patch level. Unlike prior methods that perform a single, static retrieval before generation and condition the entire generation on fixed reference images, AR-RAG performs context-aware retrievals at each generation step, using prior-generated patches as queries to retrieve and incorporate the most relevant patch-level visual references, enabling the model to respond to evolving generation needs while avoiding limitations (e.g., over-copying, stylistic bias, etc.) prevalent in existing methods. To realize AR-RAG, we propose two parallel frameworks: (1) Distribution-Augmentation in Decoding (DAiD), a training-free plug-and-use decoding strategy that directly merges the distribution of model-predicted patches with the distribution of retrieved patches, and (2) Feature-Augmentation in Decoding (FAiD), a parameter-efficient fine-tuning method that progressively smooths the features of retrieved patches via multi-scale convolution operations and leverages them to augment the image generation process. We validate the effectiveness of AR-RAG on widely adopted benchmarks, including Midjourney-30K, GenEval and DPG-Bench, demonstrating significant performance gains over state-of-the-art image generation models.

Figures

Figures reproduced from arXiv: 2506.06962 by the authors.

Figure 1
Figure 1. Comparison between Autoregressive Retrieval Augmentation ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The decoding process in Distribution-Augmentation in Decoding (DAiD). [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overall architecture of Feature-Augmentation in Decoding (FAiD). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative results of DAiD, FAiD and baselines. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Images generated by ImageRAG [33] and our AR-RAG. ImageRAG excessively copies retrieved images and does not follow user prompts [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: l2 distance between ground-truth tokens and top-10 retrieved tokens (blue line) compared to randomly sampled tokens (red dashed line). The curved arrow indicates a broken y-axis that accom￾modates the large gap between the retrieved token and the random token baseline.…
Figure 7
Figure 7. Figure 7: Hyperparameter optimization results for DAiD and FAiD on FID scores. Left: FID [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Spectral Origins of the Self-Correction Blind Spot in Autoregressive Generation

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Self-correction blind spots in residual-stream autoregressive models arise iff the product of attention Jacobians has spectral radius ≥1, with a sharp marker threshold and RL coupling condition derived from that radius.

Reference graph

Works this paper leans on

50 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    Towards effective long conversation generation with dynamic topic tracking and recommendation

    Trevor Ashby, Adithya Kulkarni, Jingyuan Qi, Minqian Liu, Eunah Cho, Vaibhav Kumar, and Lifu Huang. Towards effective long conversation generation with dynamic topic tracking and recommendation. In Proceedings of the 17th International Natural Language Generation Conference, pages 540–556, 2024

    work page 2024

  2. [2]

    Deepseek llm: Scaling open-source language models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024

    arXiv 2024

  3. [3]

    Semi- parametric neural image synthesis

    Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Semi- parametric neural image synthesis. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022

    work page 2022

  4. [4]

    Improving language models by retrieving from trillions of tokens

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, ...

    work page 2022

  5. [5]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.CoRR, abs/2102.08981, 2021

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.CoRR, abs/2102.08981, 2021

    arXiv 2021

  6. [6]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

    work page 2025

  7. [7]

    Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. CoRR, abs/2310.00426, 2023

    arXiv

  8. [8]

    Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W. Cohen. Re-imagen: Retrieval- augmented text-to-image generator. In The Eleventh International Conference on Learning Representations, 2023

    work page 2023

Show all 50 references
  1. [9]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. CoRR, abs/2501.17811, 2025

    2025 arXiv

  2. [10]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthes...

    2024

  3. [11]

    Frido: Feature pyramid diffusion for complex scene image synthesis

    Wan-Cyuan Fan, Yen-Chun Chen, Dongdong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. CoRR, abs/2208.13753, 2022

    2022 arXiv

  4. [12]

    Datacomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussm...

    2023

  5. [13]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. CoRR, abs/2312.10997, 2023

    2023 arXiv

  6. [14]

    Geneval: An object-focused framework for evaluating text-to-image alignment

    Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. CoRR, abs/2310.11513, 2023

    2023 arXiv

  7. [15]

    Kat: A knowledge augmented transformer for vision-and-language

    Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. Kat: A knowledge augmented transformer for vision-and-language. arXiv preprint arXiv:2112.08614, 2021

    2021 arXiv

  8. [16]

    Retrieval augmented language model pre-training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Le...

    2020

  9. [17]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. InProceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, p...

    2017

  10. [18]

    Ella: Equip diffusion models with llm for enhanced semantic alignment, 2024

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment, 2024

    2024

  11. [19]

    Leveraging passage retrieval with generative models for open domain question answering

    Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computationa...

    2021

  12. [20]

    Rethinking fid: Towards a better evaluation metric for image generation

    Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In CVPR, pages 9307–9315, 2024

    2024

  13. [21]

    Billion-scale similarity search with GPUs

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019

    2019

  14. [22]

    Humansd: A native skeleton-guided diffusion model for human image generation

    Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. Humansd: A native skeleton-guided diffusion model for human image generation. CoRR, abs/2304.04269, 2023

    2023 arXiv

  15. [23]

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Hugo Laroche...

    2020

  16. [24]

    Fine-grained late- interaction multi-modal retrieval for retrieval augmented visual question answering

    Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained late- interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems, 36:22820–22840, 2023

    2023

  17. [25]

    Preflmr: Scaling up fine-grained late-interaction multi-modal retrievers

    Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. Preflmr: Scaling up fine-grained late-interaction multi-modal retrievers. arXiv preprint arXiv:2402.08327, 2024

    2024 arXiv

  18. [26]

    Handrefiner: Refining malformed hands in generated images by diffusion-based conditional inpainting

    Wenquan Lu, Yufei Xu, Jing Zhang, Chaoyue Wang, and Dacheng Tao. Handrefiner: Refining malformed hands in generated images by diffusion-based conditional inpainting. In Jianfei Cai, Mohan S. Kankanhalli, Balakrishnan Prabhakaran, Susanne Boll, Ramanathan Subramanian, 11 Liang ...

    2024

  19. [27]

    Transfer between modalities with metaqueries

    Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Ji Hou, and Saining Xie. Transfer between modalities with metaqueries. CoRR, abs/2504.06256, 2025

    2025 arXiv

  20. [28]

    SDXL: improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vi...

    2024

  21. [29]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. pages 8748–8763, 2021

    2021

  22. [30]

    Hierarchical text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022

    2022 arXiv

  23. [31]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. CoRR, abs/2112.10752, 2021

    2021 arXiv

  24. [32]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022

    2022

  25. [33]

    Bermano, and Ohad Fried

    Rotem Shalev-Arkushin, Rinon Gal, Amit H. Bermano, and Ohad Fried. Imagerag: Dynamic image retrieval for reference-guided image generation, 2025

    2025

  26. [34]

    kNN-diffusion: Image generation via large-scale retrieval

    Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. kNN-diffusion: Image generation via large-scale retrieval. In The Eleventh International Conference on Learning Representations, 2023

    2023

  27. [35]

    Retrieval aug- mentation reduces hallucination in conversation

    Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval aug- mentation reduces hallucination in conversation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Findings of the Association for Computational Linguistics...

    2021

  28. [36]

    Journeydb: A benchmark for generative image understanding

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li. Journeydb: A benchmark for generative image understanding. In Alice Oh, Tristan Naumann, Amir Globerson, Kat...

    2023

  29. [37]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. CoRR, abs/2406.06525, 2024

    2024 arXiv

  30. [38]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Pr...

    2024

  31. [39]

    Fréchet wavelet distance: A domain-agnostic metric for image generation

    Lokesh Veeramacheneni, Moritz Wolter, Hilde Kuehne, and Juergen Gall. Fréchet wavelet distance: A domain-agnostic metric for image generation. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  32. [40]

    Midjourney prompts dataset

    Vivym. Midjourney prompts dataset. https://huggingface.co/datasets/vivym/ midjourney-prompts, 2023. Accessed: 2024-04-11. 12

    2023

  33. [41]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tie...

    2024 arXiv

  34. [42]

    Uniir: Training and benchmarking universal multimodal information retrievers

    Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. arXiv preprint arXiv:2311.17136, 2023

    2023 arXiv

  35. [43]

    SANA: efficient high-resolution image synthesis with linear diffusion transformers

    Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. SANA: efficient high-resolution image synthesis with linear diffusion transformers. CoRR, abs/2410.10629, 2024

    arXiv

  36. [44]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. CoRR, abs/2408.12528, 2024

    2024 arXiv

  37. [45]

    Modality-specialized synergizers for interleaved vision-language generalists

    Zhiyang Xu, Minqian Liu, Ying Shen, Joy Rimchala, Jiaxin Zhang, Qifan Wang, Yu Cheng, and Lifu Huang. Modality-specialized synergizers for interleaved vision-language generalists. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, Apr...

    2025

  38. [46]

    Retrieval-augmented multimodal language modeling

    Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-Tau Yih. Retrieval-augmented multimodal language modeling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and...

    2023

  39. [47]

    Making retrieval-augmented language models robust to irrelevant context

    Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. CoRR, abs/2310.01558, 2023

    2023 arXiv

  40. [48]

    Scaling autoregressive multi-modal models: Pretraining and instruction tuning

    Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wen Li, Susan Zhang, R...

    2023 arXiv

  41. [49]

    FineRAG: Fine-grained retrieval-augmented text-to-image generation

    Huaying Yuan, Ziliang Zhao, Shuting Wang, Shitao Xiao, Minheng Ni, Zheng Liu, and Zhicheng Dou. FineRAG: Fine-grained retrieval-augmented text-to-image generation. In Owen Ram- bow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, ed...

    2025

  42. [2021]

    Association for Computational Linguistics

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.