Pith. sign in

REVIEW 3 major objections 5 minor 50 references

Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RAG has an intrinsic generalization ceiling that plain in-context learning avoids.

desk verdict Genuinely new finite-sample bounds for RAG as noisy ICL, with a real but repairable gap in the headline 'intrinsic ceiling' claim. read the letter →

arxiv 2506.03100 v3 pith:HAHC26NZ submitted 2025-06-03 cs.LG cs.AIcs.CLcs.IRmath.STstat.TH

Yang Guo , Yutian Tao , Yifei Ming , Robert D. Nowak , Yingyu Liang This is my paper
classification cs.LGcs.AIcs.CLcs.IRmath.STstat.TH MSC 68T0762J05
keywords retrieval-augmentedgenerationin-contextlearningbias-variancetradeofflinearregressiontransformertheorygeneralizationboundsnoisydemonstrationsself-attention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-augmented generation (RAG) has been shown to work well in practice, but a theory of how retrieved examples shape the generalization error has been missing. This paper supplies one for in-context linear regression with a single-layer linear self-attention model: it treats each retrieved example as a query-dependent noisy copy of the query, derives the exact bias-variance decomposition of the prediction risk, and shows that the variance component shrinks as the number of retrieved examples grows while the bias component does not. Consequently, RAG has an intrinsic ceiling: beyond a certain number of retrieved examples, the error stops improving and can even worsen. The framework recovers standard in-context learning and standard RAG as limit cases, and predicts an optimal retrieval count that grows slowly with the number of in-context examples. These predictions match experiments on Natural Questions and TriviaQA, where the first few retrieved examples help and later ones hurt.

What carries the argument

The load-bearing object is the single-layer linear self-attention (LSA) predictor $\hat{y}_q = x_q^\top W X^\top y$, together with the retrieval-noise model $x_i^{\rm rag} = x_q + r_i$ with $r_i \sim N(0, \delta_i^2 I_d)$. The retrieval offsets make $X^{\rm rag}$ depend on the query, which forces the bias computation to evaluate a 6th-order Gaussian monomial; the paper derives that expectation by enumerating all 15 perfect matchings via Isserlis' theorem (Lemma 3). The bias-variance split comes from conditioning on the weight $W$ and using the scaling $W = \frac{m}{m+n}\bar{W}$ to adapt the pretrained weight to the longer context.

What would settle it

Measure exact-match accuracy on a QA benchmark for a fixed RAG model as $n$ takes values 1, 2, 4, 8, 16, 32, 64, and check whether the error curve flattens at a positive level rather than continuing toward the plain-ICL level. If the error keeps decreasing to the ICL baseline as $n$ grows, the intrinsic-ceiling claim fails. A second check: estimate the distribution of $r_i = x_i^{\rm rag} - x_q$ on real retrieval outputs; if the distribution is clearly non-Gaussian, the paper's quantitative predictions are not applicable, though the plateau could still appear.

Watch

Extended reading notes

Core claim

The paper claims that in the in-context linear regression setting, the test-time loss of RAG decomposes exactly into a variance term that vanishes with the number of retrieved examples $n$ and a bias term that does not. As $n\to\infty$ with $m$ fixed, the variance-induced error decays as $O(1/n)$ but the bias-induced error plateaus at a positive constant of order $d^2\|\beta_{tt}\|_2^2$, so the total risk cannot go below a model-dependent floor. This is the "intrinsic ceiling" that distinguishes RAG from standard ICL, where bias also decays with the context size. Under uniform retrieval noise, the optimal number of RAG examples $n^*$ is finite and independent of $m$ to first order, and the best improvement over no RAG shrinks as $O(1/m^2)$. Under non-uniform noise, where noise grows with retrieval distance, adding more retrieved examples can cause the variance error to diverge if the retrieval quality decays too fast.

Load-bearing premise

The whole derivation rests on assuming each retrieved example is the query embedding plus Gaussian noise with a variance that does not depend on the query; if real retrieval offsets are non-Gaussian or depend on the query in other ways, the exact ceiling result is not established.

Editorial extensions

If this is right

  • Given a fixed number $m$ of in-context examples, the optimal number of retrieved examples $n^*$ is finite; adding more than $n^*$ retrieved examples increases rather than decreases the risk.
  • The gain from optimally choosing $n$ over using no RAG at all shrinks like $O(1/m^2)$, so RAG's benefit is largest when $m$ is small.
  • Under non-uniform retrieval noise with power-law decay, the variance error diverges for decay exponents $q>1/2$ (distance-proportional noise) or $q>1$ (mixture noise), predicting a sharp regime where extra retrieval is actively harmful.
  • The theory predicts that retrieval from a large external corpus behaves like uniform noise (slow monotone gains), while retrieval from a small training set transitions from uniform to non-uniform noise as more examples are added, matching the ATLAS/RAVEN experiments against Natural Questions and TriviaQA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable design rule follows: measure the empirical variance $\delta_i^2$ of retrieved embeddings around the query and stop adding retrieved documents once the predicted $n^*$ is reached; this gives a principled early-stopping criterion for RAG pipelines.
  • If the Gaussian-offset assumption fails (e.g., heavy-tailed embedding differences), the exact rates may change, but the qualitative ceiling should persist for any retrieval distribution with comparable variance, suggesting the plateau is structural rather than an artifact of Gaussianity.
  • The uniform versus non-uniform distinction maps retrieval from closed versus open universes: training-set retrieval becomes non-uniform once the near neighbors are exhausted, while Wikipedia-scale corpora remain uniform for longer; this could be formalized as a pool-size effect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a unified theoretical treatment of retrieval-augmented generation (RAG) as noisy in-context learning (ICL). It models each retrieved example as the query vector plus a Gaussian offset, x_i^rag = x_q + r_i with r_i ~ N(0, delta_i^2 I), and studies a single-layer linear self-attention predictor under MSE loss. For uniform retrieval noise, Theorem 1 gives an exact bias-variance decomposition of the population loss, Proposition 3 gives a closed form for the isotropically trained weight W*, and Proposition 1 gives asymptotic rates. The paper claims that RAG has an intrinsic ceiling on generalization error, in contrast to ICL, because the bias-induced error plateaus as the number of retrieved examples n grows. For non-uniform noise, Theorems 2 and 3 analyze distance-proportional and distance-weighted mixture noise under a power-law growth of delta_i^2. Experiments on Natural Questions and TriviaQA with ATLAS and RAVEN are presented as qualitative support.

Significance. If the central claims are made rigorous, this is a valuable contribution: it provides the first exact finite-n risk decomposition for RAG in in-context linear regression, recovers classical ICL at n=0 as a sanity check, and offers a concrete mechanism (bias saturation) for the empirically observed degradation of RAG when too many retrieved examples are added. The proof of Theorem 1 is self-contained and carefully executed, and Proposition 3 is a useful exact characterization. The paper also gives a falsifiable qualitative prediction about an optimal retrieval count. However, the 'intrinsic ceiling' claim as stated is not established by the provided upper bounds, and the data model is internally ambiguous. These issues are repairable but require substantive additions before the paper can be accepted.

major comments (3)
  1. [§4.1, Proposition 1 vs. Proposition 3] The abstract and Section 4.1 claim that RAG has an 'intrinsic ceiling' because the bias-induced error plateaus as n grows, but Proposition 1 only bounds err_bias(W*) from above: for n→∞ with m fixed, the displayed O(||β_tt||^2 d^2) is compatible with err_bias → 0. A plateau requires a liminf lower bound, which is not stated or proved. The exact expression in Proposition 3 does allow the limit to be computed: taking n→∞ in the displayed formula gives ||β_tt||^2 [1 − 2m(δ^2+d+2)/(m+d+1) + m^2(d^2+6d+8+4δ^2+2dδ^2+δ^4)/(m+d+1)^2], which I find to be positive for the parameter values I checked. The authors should state this as an explicit lower-bound corollary, and similarly for the n=Θ(m) row of Eq. (13), before using the ceiling language in the abstract and Section 4.1.
  2. [Assumption 2.3 and Section 3] The data model is internally inconsistent. Section 3 defines D_rag(x_q) by x_i^rag = x_q + r_i with r_i ~ N(0, δ_i^2 I) (Assumption 1), but Assumption 2.3 states that x_i^rag are i.i.d. N(0,I) for all RAG indices. These are incompatible: the marginal of x_q + r_i is not N(0,I). The proof of Theorem 1 in Appendix B.1, especially Eqs. (50)–(56), relies on the query-centered construction, so the exact decomposition in Eq. (11) does not follow under a literal reading of Assumption 2.3. Moreover, if Assumption 2.3 were taken literally, the RAG blocks would act as additional i.i.d. ICL examples and the query-dependent source of bias central to the ceiling claim would be removed. The authors should revise Assumption 2.3 to apply only to the ICL blocks and the query, and state explicitly that D_rag(x_q) is governed by Assumption 1.
  3. [Theorems 2–3] The non-uniform noise results assume δ_i^2 = γ_2 i^q with q ≥ 0, so δ_i^2 grows without bound as n increases. This is in tension with the modeling statement just before Proposition 1, where δ^2 ≪ 1 is justified by saying that it is unrealistic for retrieval variance to exceed the input variance. The paper does not delimit the validity of the power-law ansatz or the regime in which the divergence thresholds q>1/2 and q>1 are meant to apply. Since the non-uniform theorems reuse W* and the Gaussian-offset calculations from the uniform setting, the unbounded growth of δ_i^2 should be stated as a separate asymptotic model with its own domain of validity, or the bounds should be proved under a bounded-δ condition with the divergence results stated as limiting consequences.
minor comments (5)
  1. [B.1.2, Eq. (72)] The displayed second-derivative expression in Eq. (72) is not the correct derivative of Eq. (71); the subsequent monotonicity argument in the same proof is sufficient for the global-minimizer claim, but the incorrect display should be corrected.
  2. [Assumption 5] The text says the constant 1 in p_i = (1+δ_i^2)^{-q̃} 'ensures p_i = 0 when δ_i^2 = 0', but the formula gives p_i = 1 when δ_i^2 = 0. This should be corrected.
  3. [Theorem 3] Theorem 3 mixes hat and tilde notation for the variance-induced error: the statement uses ˆ err_variance where the mixture-noise variance error should be ˜ err_variance. Please make the notation consistent.
  4. [Section 5] The experiments are qualitative and do not estimate the retrieval offsets δ_i, fit the power-law exponent q, or compare against the predicted optimal n*, so they should be described as illustrative evidence for the theory rather than quantitative validation of the bounds.
  5. [General] Proposition 3, which contains the exact closed-form loss that is needed to verify the ceiling claim, appears only in the appendix; moving it to the main text (or at least stating the n→∞ lower-bound corollary in the main text) would substantially improve the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the RAG error derivation is self-contained and the ceiling claim, while open to a correctness objection, is not a circular reduction.

full rationale

The paper's central derivation (Theorem 1, Proposition 3, Proposition 1) computes the population loss of a linear self-attention predictor under explicit distributional assumptions (Assumptions 1-3) using Isserlis' theorem and Gaussian moment identities. No parameter is fitted to the QA experiments and then renamed a prediction; the experimental section only qualitatively aligns observed trends with the noise regimes in the theory. The n=0 recovery of the ICL loss is a sanity check, not an input. The W* = m/((m+d+1)(m+n))I form is introduced as an explicit special case (Proposition 3), and the bounds are derived, not assumed. There are no load-bearing self-citations: the cited prior work (Ahn et al., Zhang et al., Gozeten et al.) supplies the LSA framework and the ICL loss, but the RAG-specific terms are computed in this paper. The 'intrinsic ceiling' wording is a potential logical gap because Proposition 1 gives an upper bound on err_bias while a plateau is a lower-bound claim, and Assumption 2.3 appears inconsistent with the query-centered Assumption 1; these are correctness and consistency concerns, not circularity. No step reduces to its own input by construction.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The analysis introduces no new physical entities. The free parameters are all the auxiliary scales and exponents of the two ad hoc non-uniform noise models (DPN and mixture). The core uniform-noise result depends only on the standard problem parameters m, n, d, sigma^2, sigma_rag^2, and delta^2, all supplied by the data model rather than fitted.

free parameters (5)
  • gamma_2 (DPN power-law scale)
    delta_i^2 = gamma_2 i^q, introduced ad hoc in Theorem 2 to model distance-proportional noise; controls when variance error diverges.
  • q (DPN power-law exponent)
    Exponent in delta_i^2 = gamma_2 i^q; determines divergence threshold q>1/2 for variance and q>0 for bias. Chosen to illustrate heavy-tailed retrieval noise, not derived.
  • gamma_1 (DPN noise scale)
    sigma_rag,i^2 = gamma_1 sigma^2 delta_i^2, ad hoc coupling of label noise to retrieval distance.
  • c_s and c_l (mixture noise scales)
    Small and large label noise variances in Assumption 5; ad hoc to model correct and incorrect retrieved passages.
  • tilde_q (mixture decay exponent)
    p_i = (1+delta_i^2)^{-tilde_q}, ad hoc power-law decay of the probability that a retrieved item has small noise.
assumptions (6)
  • domain assumption Gaussian retrieval offset: r_i ~ N(0, delta_i^2 I_d) for each retrieved example.
    Assumption 1 in Section 4. The whole bias-variance decomposition relies on Gaussianity of the offset.
  • domain assumption Isotropic Gaussian data and weights: x_i, x_q ~ N(0,I), beta ~ N(0,I), label noise Gaussian.
    Assumption 2 in Section 4. Required for closed-form Gaussian moments.
  • domain assumption Single-layer linear self-attention parameterization with W_Q W_K^T and W_V fixed.
    Equation (10). The authors cite prior ICL theory that this captures ICL, but it does not cover depth or nonlinearity.
  • domain assumption Pretrained weight W* = m / ((m+d+1)(m+n)) I.
    Proposition 3. The ceiling result depends on W* being isotropic and independent of n except through the m/(m+n) scaling.
  • domain assumption delta^2 << 1 in Proposition 1.
    Stated before Proposition 1. Justified as 'unrealistic to assume a higher retrieval variance than the input variance', but this smallness is silently abandoned in the non-uniform theorems.
  • ad hoc to paper Power-law growth delta_i^2 = gamma_2 i^q in Theorems 2 and 3.
    Assumptions 4 and 5. Introduces a specific heavy-tailed retrieval noise model; the divergence results are specific to this choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds." pith.science (2026). https://pith.science/paper/HAHC26NZ

@misc{pith2026250603100,
  author       = {Pith},
  title        = {Pith review of: Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HAHC26NZ}},
  note         = {Machine review of arXiv:2506.03100}
}
read the original abstract

Retrieval-augmented generation (RAG) has seen many empirical successes in recent years by aiding the LLM with external knowledge. However, its theoretical aspect has remained mostly unexplored. In this paper, we propose the first finite-sample generalization bound for RAG in in-context linear regression and derive an exact bias-variance tradeoff. Our framework views the retrieved texts as query-dependent noisy in-context examples and recovers the classical in-context learning (ICL) and standard RAG as the limit cases. Our analysis suggests that an intrinsic ceiling on generalization error exists on RAG as opposed to the ICL. Furthermore, our framework is able to model retrieval both from the training data and from external corpora by introducing uniform and non-uniform RAG noise. In line with our theory, we show the sample efficiency of ICL and RAG empirically with experiments on common QA benchmarks, such as Natural Questions and TriviaQA.

Figures

Figures reproduced from arXiv: 2506.03100 by the authors.

Figure 1
Figure 1. We compare performance between the RAG-only ( [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. We compare the performance of RAG using examples ( [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Performance sensitivity to the ratio n1/n under different data points c, where n1 refers to retrieved examples and n2 to passages. References Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36:45614–45650, 2023. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Han… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 19 canonical work pages

  1. [1]

    Transformers learn to implement preconditioned gradient descent for in-context learning

    Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36: 0 45614--45650, 2023

    2023

  2. [2]

    Self-RAG : Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG : Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023

    arXiv 2023

  3. [3]

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, T. W. Hennigan, Saffron Huang, Lorenzo Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving,...

    work page 2021

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  5. [5]

    Reading wikipedia to answer open-domain questions

    Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051, 2017

    arXiv 2017

  6. [6]

    Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Annual Meeting of the Association for Computational Linguistics, 2024

    work page 2024

  7. [7]

    Exploring the robustness of in-context learning with noisy labels

    Chen Cheng, Xinzhi Yu, Haodong Wen, Jingsong Sun, Guanzhang Yue, Yihao Zhang, and Zeming Wei. Exploring the robustness of in-context learning with noisy labels. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE, 2025

    work page 2025

  8. [8]

    Uprise: Universal prompt retrieval for improving zero-shot evaluation

    Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Denvy Deng, and Qi Zhang. Uprise: Universal prompt retrieval for improving zero-shot evaluation. arXiv preprint arXiv:2303.08518, 2023

    arXiv 2023

Show all 50 references
  1. [9]

    Nearest neighbor pattern classification

    Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13 0 (1): 0 21--27, 1967

    1967

  2. [10]

    A survey for in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022

    2022 arXiv

  3. [11]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages...

    2024

  4. [12]

    What can transformers learn in-context? a case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35: 0 30583--30598, 2022

    2022

  5. [13]

    Test-time training provably improves transformers as in-context learners

    Halil Alperen Gozeten, M Emrullah Ildiz, Xuechen Zhang, Mahdi Soltanolkotabi, Marco Mondelli, and Samet Oymak. Test-time training provably improves transformers as in-context learners. arXiv preprint arXiv:2503.11842, 2025

    2025

  6. [14]

    Raven: In-context learning with retrieval-augmented encoder-decoder language models, 2024

    Jie Huang, Wei Ping, Peng Xu, Mohammad Shoeybi, Kevin Chen-Chuan Chang, and Bryan Catanzaro. Raven: In-context learning with retrieval-augmented encoder-decoder language models, 2024. URL https://arxiv.org/abs/2308.07922

    2024 arXiv

  7. [15]

    Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models

    Rongjie Huang, Jia-Bin Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiaoyue Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. ArXiv, abs/2301.12661, 2023

    2023 arXiv

  8. [16]

    On a formula for the product-moment coefficient of any order of a normal frequency distribution in any number of variables

    Leon Isserlis. On a formula for the product-moment coefficient of any order of a normal frequency distribution in any number of variables. Biometrika, 12 0 (1/2): 0 134--139, 1918

    1918

  9. [17]

    Leveraging passage retrieval with generative models for open domain question answering

    Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. ArXiv, abs/2007.01282, 2020

    2007 arXiv

  10. [18]

    Atlas: Few-shot learning with retrieval augmented language models, 2022

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models, 2022. URL https://arxiv.org/abs/2208.03299

    2022 arXiv

  11. [19]

    T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational L...

    2017 doi

  12. [20]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  13. [21]

    Same task, more tokens: the impact of input length on the reasoning performance of large language models

    Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848, 2024

    2024 arXiv

  14. [22]

    More documents, same length: Isolating the challenge of multiple documents in rag

    Shahar Levy, Nir Mazor, Lihi Shalmon, Michael Hassid, and Gabriel Stanovsky. More documents, same length: Isolating the challenge of multiple documents in rag. arXiv preprint arXiv:2503.04388, 2025

    2025

  15. [23]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceeding...

    2020

  16. [24]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing...

    2020

  17. [25]

    Making large language models a better foundation for dense retrieval

    Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. Making large language models a better foundation for dense retrieval. ArXiv, abs/2312.15503, 2023

    2023

  18. [26]

    Self-prompting large language models for zero-shot open-domain qa

    Junlong Li, Jinyuan Wang, Zhuosheng Zhang, and Hai Zhao. Self-prompting large language models for zero-shot open-domain qa. arXiv preprint arXiv:2212.08635, 2022

    2022 arXiv

  19. [27]

    Mot: Pre-thinking and recalling enable chatgpt to self-improve with memory-of-thoughts

    Xiaonan Li and Xipeng Qiu. Mot: Pre-thinking and recalling enable chatgpt to self-improve with memory-of-thoughts. CoRR, 2023

    2023

  20. [28]

    Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources

    Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. In The Twelfth International Conference on Learning Representations, 2024

    2024

  21. [29]

    What makes good in-context examples for gpt- 3 ? arXiv preprint arXiv:2101.06804, 2021

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt- 3 ? arXiv preprint arXiv:2101.06804, 2021

    2021 arXiv

  22. [30]

    In-context learning with retrieved demonstrations for language models: A survey

    Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. In-context learning with retrieved demonstrations for language models: A survey. arXiv preprint arXiv:2401.11624, 2024

    2024 arXiv

  23. [31]

    Z-icl: Zero-shot in-context learning with pseudo-demonstrations

    Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi. Z-icl: Zero-shot in-context learning with pseudo-demonstrations. arXiv preprint arXiv:2212.09865, 2022

    2022 arXiv

  24. [32]

    Sfr-embedding-mistral:enhance text retrieval with transfer learning

    Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embedding-mistral:enhance text retrieval with transfer learning. Salesforce AI Research Blog, 2024. URL https://blog.salesforceairesearch.com/sfr-embedded-mistral/

    2024

  25. [33]

    Metaicl: Learning to learn in context

    Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943, 2021

    2021 arXiv

  26. [34]

    The matrix cookbook

    Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark, 7 0 (15): 0 510, 2008

    2008

  27. [35]

    In-context retrieval-augmented language models

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11: 0 1316--1331, 2023

    2023

  28. [36]

    Smallcap: Lightweight image captioning prompted with retrieval augmentation

    Rita Parada Ramos, Bruno Martins, Desmond Elliott, and Yova Kementchedjhieva. Smallcap: Lightweight image captioning prompted with retrieval augmentation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2840--2849, 2022

    2023

  29. [37]

    Retrieval-augmented transformer for image captioning

    Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Retrieval-augmented transformer for image captioning. Proceedings of the 19th International Conference on Content-based Multimedia Indexing, 2022

    2022

  30. [38]

    Xricl: Cross-lingual retrieval-augmented in-context learning for cross-lingual text-to-sql semantic parsing

    Peng Shi, Rui Zhang, He Bai, and Jimmy Lin. Xricl: Cross-lingual retrieval-augmented in-context learning for cross-lingual text-to-sql semantic parsing. arXiv preprint arXiv:2210.13693, 2022

    2022 arXiv

  31. [39]

    Leave no document behind: Benchmarking long-context llms with extended multi-doc qa

    Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. arXiv preprint arXiv:2406.17419, 2024 a

    2024 arXiv

  32. [40]

    Can in-context learning really generalize to out-of-distribution tasks? arXiv preprint arXiv:2410.09695, 2024 b

    Qixun Wang, Yifei Wang, Yisen Wang, and Xianghua Ying. Can in-context learning really generalize to out-of-distribution tasks? arXiv preprint arXiv:2410.09695, 2024 b

    2024 arXiv

  33. [41]

    Certifiably robust rag against retrieval corruption

    Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. Certifiably robust rag against retrieval corruption. arXiv preprint arXiv:2405.15556, 2024

    2024

  34. [42]

    An explanation of in-context learning as implicit bayesian inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080, 2021

    2021 arXiv

  35. [43]

    RECOMP : Improving retrieval-augmented LM s with context compression and selective augmentation

    Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP : Improving retrieval-augmented LM s with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations, 2024

    2024

  36. [44]

    Is retriever merely an approximator of reader? arXiv preprint arXiv:2010.10999, 2020

    Sohee Yang and Minjoon Seo. Is retriever merely an approximator of reader? arXiv preprint arXiv:2010.10999, 2020

    2010 arXiv

  37. [45]

    Compositional exemplars for in-context learning

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional exemplars for in-context learning. In International Conference on Machine Learning, pages 39818--39833. PMLR, 2023

    2023

  38. [46]

    Making retrieval-augmented language models robust to irrelevant context

    Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. arXiv preprint arXiv:2310.01558, 2023

    2023 arXiv

  39. [47]

    Trained transformers learn linear models in-context

    Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25 0 (49): 0 1--55, 2024

    2024

  40. [48]

    Automatic chain of thought prompting in large language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022

    2022 arXiv

  41. [49]

    Retrieval-augmented generation for ai-generated content: A survey

    Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.19473, 2024 a

    2024 arXiv

  42. [50]

    Noisyicl: A little noise in model parameters calibrates in-context learning

    Yufeng Zhao, Yoshihiro Sakai, and Naoya Inoue. Noisyicl: A little noise in model parameters calibrates in-context learning. arXiv preprint arXiv:2402.05515, 2024 b

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.