Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

The Role of Diversity in In-Context Learning for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that diversity-aware demonstration selection improves in-context learning on challenging tasks and out-of-distribution queries, and provides a linear-regression theory of why.

desk verdict Big empirical survey of diversity in ICL demonstration selection, with a theory that does not actually cover the regime it is invoked to explain. read the letter →

arxiv 2505.19426 v2 pith:JB56O6EA submitted 2025-05-26 cs.CL cs.AIcs.LG

Wenyang Xiao , Haoyu Zhao , Lingxiao Huang This is my paper
classification cs.CLcs.AIcs.LG
keywords in-contextlearningdemonstrationselectiondiversitysimilarity-basedretrievalout-of-distributiongeneralizationLLMpromptinglinearregressiontheory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that when choosing which examples to put in an LLM prompt, favoring diversity among the demonstrations — not just similarity to the query — improves in-context learning on demanding tasks and on out-of-distribution queries. Across sentiment, reasoning, reading comprehension, math, and SQL-code benchmarks on Llama-3.1, Gemma-2, and Mistral-v0.3, diversity-aware selection (Div and TopK-Div) beats similarity-only TopK on math, code, and reading comprehension, while TopK remains best on simple sentiment. The paper also reports that diversity helps most on individually harder examples and when demonstrations come from a different distribution than queries. A linear-regression model with binary "skill" embeddings is used to show why: diverse demonstrations can cover the query's required skills, and can help even when coverage is already complete.

What carries the argument

The load-bearing object is the TopK-Div selection rule: examples are chosen greedily to maximize α·Similarity(E(x), query) + (1−α)·Diversity(E(x), already-selected set). Div is the special case α=0 after building a diverse "coreset". The theoretical mechanism is a linear-regression model of ICL in which the LLM is assumed to compute the min-norm solution over the selected demonstrations (Assumption 4.1), with inputs encoded as binary vectors whose 1-entries are "skills" or local structures. Two constructed distributions then show TopK-Div has lower expected loss than TopK: one where diversity helps by covering the query's non-zero entries (coverage), and one where both methods fully cover the query yet diversity still wins (beyond coverage).

What would settle it

Run the same comparison on sentiment or other simple tasks reformatted to be as generation-heavy as math and code, for example multi-step sentiment reasoning with exact-match grading: if TopK still beats TopK-Div under that controlled complexity, the complexity-based explanation of the gains would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that diversity in demonstration selection is a genuine performance lever for in-context learning, not a side effect of coverage: explicitly mixing similarity with diversity (TopK-Div, with α=1/2) or selecting from a diverse coreset (Div) consistently improves over pure TopK on harder tasks, on harder examples, and under distribution shift, across open-weight model families and scales from 1B to 70B. On simple sentiment classification the ordering reverses, which the paper reads as evidence that task complexity controls when diversity pays. The paper further claims a "beyond coverage" effect: diversity helps even when all of a query's relevant features are already covered by the selected demonstrations, and it supports this claim with SQuAD noise-trimming experiments and with Theorem 4.3 for α≥1−1/l.

Load-bearing premise

That the difference between "easy" and "hard" tasks is really task complexity rather than confounds like output format, answer length, or dataset noise, and that per-example difficulty can be measured by a fine-tuned model's zero-shot accuracy.

Editorial extensions

If this is right

  • On text-to-SQL (GeoQuery), TopK-Div outperforms TopK by at least 7 absolute accuracy points in the standard split and by larger margins in out-of-distribution splits.
  • On math (GSM8K, GSM-Plus-Mini, PRM800K), diversity-aware methods match or beat TopK, with the largest gains when the query set is out-of-distribution.
  • On reading comprehension, diversity-aware methods beat TopK, and the benefit concentrates on examples a fine-tuned model cannot answer zero-shot.
  • Scaling models from 1B to 70B does not erase the diversity advantage on GeoQuery and SCIQ, so the effect is not limited to small models.
  • Random and k-means selections can beat TopK on math and reading, showing the diversity advantage is not tied to one specific implementation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If task complexity is the true driver, diversity-aware selection should also help on other structured generation tasks such as formal proofs or tool-use planning, which the paper does not test.
  • The beyond-coverage result suggests a testable extension: corrupting support passages with noise should degrade TopK more than TopK-Div, because diversity may protect against overfitting to noisy surface patterns.
  • The fixed defaults (α=1/2 and Div coreset size 100) are likely dataset-dependent, and an adaptive α that rises with estimated task difficulty or distribution shift could yield further gains beyond the paper's experiments.
  • The theory assumes binary skill embeddings and min-norm regression; extending it to non-binary or compositional skill vectors could predict the boundary where added diversity stops helping.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates whether diversity-aware selection of in-context examples improves LLM performance relative to similarity-only selection. It introduces TopK-Div, a method that greedily balances query similarity and pairwise diversity with a hyperparameter α, and compares it with random selection (Rand), TopK similarity retrieval, and a diversity-then-similarity coreset method (Div) across sentiment classification, commonsense reasoning, math, text-to-SQL, and reading comprehension, using Llama-3.1/3.2, Gemma-2, and Mistral-v0.3 models. The main empirical findings are that diversity-aware methods help on challenging tasks, on out-of-distribution queries, and on individually harder examples. The paper also develops a theoretical framework based on min-norm linear regression, with two constructed distributions (Examples I and II), proving that TopK-Div can beat TopK either by improving coverage (Theorem 4.2) or beyond coverage (Theorem 4.3).

Significance. If the empirical findings hold, the paper provides a useful and fairly broad map of when diversity in demonstration selection matters, with experiments spanning multiple model families, model sizes, shot counts, embeddings, and decoding strategies. The paper is transparent about its core assumptions, reports standard deviations for the main table in the appendix, and includes extensive ablations. The theoretical section is self-contained and the proofs are detailed, but it is a stylized illustration rather than a general theory; its value depends on how tightly it connects to the experimental claims. The main contribution is therefore the empirical characterization, and the theoretical framework is a supporting element that currently needs to be reconciled with the experimental parameter regime.

major comments (4)
  1. [Section 4 (Theorem 4.3) and Section 3 (hyperparameters)] Theorem 4.3 proves L > L' only under the condition α ≥ 1 − 1/l (with l ≥ 3, i.e., α ≥ 2/3), whereas all main experiments fix α = 1/2, and Table 3 shows the largest diversity gains on reading tasks at low α (negative Δ). The theory's 'beyond-coverage' result is therefore not instantiated in the experimental regime it is invoked to support; please either restrict the theoretical claim, add experiments in the α ≥ 2/3 regime, or provide a separate argument connecting the high-α regime to the observed low-α gains.
  2. [Table 5] The 'Average' row in Table 5 is numerically inconsistent with the per-model rows: for PRM800K, the row-by-row averages are approximately Rand 46.4, TopK 48.9, Div 46.7, TopK-Div 48.7, but the table reports 61.86, 64.97, 62.21, and 64.87. As printed, the average row shows TopK beating TopK-Div, which contradicts the text's claim that diversity-aware methods are more robust in this OOD setting; please correct the computation and re-evaluate the OOD conclusion accordingly.
  3. [Tables 4-7] Tables 4, 5, 6, and 7 report no standard deviations or confidence intervals, even though Table 10 in Appendix B shows that typical per-method stds in the main table are 0.3–1.5%. Several load-bearing differences in these tables are below 1% (e.g., Table 4, SST-2 with Amazon demonstrations: TopK 89.50 vs. TopK-Div 89.60), so without error bars it is impossible to distinguish signal from noise; please report variability or significance tests for these tables.
  4. [Section 3.1, Finding 1 and Table 1] Finding 1 states that 'Div and TopK-Div perform better' on challenging tasks, but Table 1 contains multiple cells where TopK-Div underperforms TopK on reading and math (e.g., Llama-3.1-8B SQuAD 4-shot: 73.28 vs. 73.51; GSM8K 4-shot: 81.74 vs. 81.99). The claim should be qualified to the specific method (Div) or supported by a significance test over the aggregated results, rather than asserted uniformly for both diversity-aware methods.
minor comments (5)
  1. [Title / Abstract] The title and abstract contain missing spaces ('THEROLE OFDIVERSITY ININ-CONTEXTLEARNING FOR LARGELANGUAGEMODELS'); please fix the typography.
  2. [Author affiliation] The affiliation line contains a typo: 'Nanjing Universtiy' should be 'Nanjing University'.
  3. [Section 3.1, Diversity v.s. coverage] The sentence 'their improvement margins are significantly larger than that of Div (though still more than 1% lower than Div)' is ambiguous about whether 'lower than Div' refers to absolute accuracy or improvement margin; please rephrase.
  4. [Section 3, hyperparameters / Table 3] The Δ definition in Eq. (4) implies α values 0.1, ..., 1.0, but Section 3 states α = 1/2 for main experiments; please clarify which α values are used in the Table 3 sweep and whether any high-α (≥ 2/3) settings were tested.
  5. [Appendix B.2 and Figure 4] The paragraph referencing Figure 4 says 'the relative improvement from diversity-aware selection does not diminish significantly as model size increases', but Figure 4 also includes the shot-count ablation; the caption and text should be aligned to avoid confusion.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: external-benchmark experiments and conditional theorems stand on their own; the alpha-regime mismatch is a correctness issue, not a circular step.

full rationale

The paper's central claims are empirical: on public benchmarks (SST-2, Amazon, ARC-Easy, CsQA, GSM8K, GeoQuery, SQuAD, SCIQ) it compares four selection methods across Llama, Gemma, and Mistral families, with no parameter fitted from the theory. The theoretical section is an explicit conditional model: Assumption 4.1 (min-norm ICL for linear regression) plus a binary-embedding ansatz yields Theorems 4.2 and 4.3, whose proofs in Appendix C compute expected losses for hand-specified distributions. These are genuine derivations, not restatements of the empirical findings. The only self-citation by a co-author (Zhao et al., 2025) is used as inspiration for the skill/local-structure view of embeddings; neither the theorems nor the experiments depend on that citation for their conclusions, so it is not load-bearing. The flagged mismatch between Theorem 4.3's condition alpha >= 1 - 1/l and the experimental choice alpha = 1/2 means the theorem is not instantiated in the reported experiments; this is an internal-consistency/correctness concern, not an equivalence-by-construction or a fitted-input prediction. Accordingly, no circular step is identified; the score reflects only the minor non-load-bearing self-citation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The empirical core is grounded in standard benchmarks, so the axioms listed affect mainly the theoretical justification. Assumption 4.1 and the skill-based binary embedding are explicit domain assumptions in Section 4. The two example distributions are constructed specifically to illustrate the mechanism. The hand-chosen method hyperparameters (alpha, coreset size) are ablated rather than fitted to optimize the reported numbers.

free parameters (2)
  • alpha (TopK-Div similarity-diversity trade-off) = 0.5 in main experiments; swept over [0.1, 1.0] in Table 3
    Controls weight between query similarity and demonstration diversity in Eq. (3). Chosen by hand, ablated.
  • Div coreset size m = 100
    Size of the diverse subset before TopK selection. Chosen as a compromise; Appendix B.3 shows no single value is optimal across datasets.
assumptions (4)
  • domain assumption Assumption 4.1: the LLM's ICL prediction for linear regression is the min-norm least-squares solution y_pred = <e_q, E^dagger E theta_T>.
    Used for all theorems in Section 4; cited empirical support (Akyurek et al. 2023) is for transformers trained on noiseless linear regression, not general LLMs.
  • domain assumption Each example embedding e in {0,1}^d encodes the set of skills or local structures present in the example.
    Motivates the binary embedding model in Section 4; based on skill-composition literature.
  • ad hoc to paper The distributions DE and QE in Examples I and II are chosen to represent settings where diversity helps.
    These distributions are constructed so that TopK-Div covers the query support or avoids redundant structure; they are illustrative, not derived from downstream tasks.
  • standard math Standard optimization and probability tools (Lagrangian duality, expectations over i.i.d. variables) are valid.
    Used throughout the proofs in Appendix C.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Role of Diversity in In-Context Learning for Large Language Models." pith.science (2026). https://pith.science/paper/JB56O6EA

@misc{pith2026250519426,
  author       = {Pith},
  title        = {Pith review of: The Role of Diversity in In-Context Learning for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JB56O6EA}},
  note         = {Machine review of arXiv:2505.19426}
}
read the original abstract

In-context learning (ICL) is a crucial capability of current large language models (LLMs), where the selection of examples plays a key role in performance. While most existing approaches focus on selecting the most similar examples to the query, the impact of diversity in example selection remains underexplored. We systematically investigate the role of diversity in in-context example selection through experiments across a range of tasks, from sentiment classification to more challenging math and code problems. Experiments on Llama-3.1, Gemma-2, and Mistral-v0.3 families of models show that diversity-aware selection methods improve performance, particularly on complex tasks like math and code, and enhance robustness to out-of-distribution queries. To support these findings, we introduce a theoretical framework that explains the benefits of incorporating diversity in in-context example selection.

Figures

Figures reproduced from arXiv: 2505.19426 by the authors.

Figure 1
Figure 1. An illustrative example for TopK, Div, and TopK-Div methods. Point filled in black denotes the query. (a) TopK: Select the most similar demonstrations (3 points filled in red) in the embedding space. (b) Div: First select a “coreset” based on some diversity metric, which is fixed for all queries (6 points filled in gray or red). Then select the most similar demonstrations (3 points filled in red) among this “coreset… view at source ↗
Figure 2
Figure 2. (Comparison of different methods on GeoQeury OOD setting) We report the relative improvement of TopK-Div over TopK when demonstrations and queries come from different GeoQuery dataset splitting ways. “standard” split denotes ID the setting. The relative improvement enlarges in the OOD setting. Text-to-SQL generation follows this pattern, with a larger improvement in OOD settings. Additionally, we note that GSM-Plus-… view at source ↗
Figure 3
Figure 3. The relative improvement of diversity-aware methods over TopK. Left: relative improvement of TopK-Div over TopK on GeoQuery standard split. Right: relative improvement of Div over TopK on SCIQ. and TopK-Div. Notably, their improvement margins are significantly larger than that of Div (though still more than 1% lower than Div). This indicates that when the dataset quality is higher, the “Coverage” mechanism can focus… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The performance of different demonstration selection methods with different number of shots [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: (The accuracy of different methods with different number of shots K on reading comprehension tasks.) We choose report the results on Llama-3.1-8B, with Sentence-BERT embeddings (all-roberta-large-v1). Left: results where demonstration and query come from SCIQ. Right: r…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning to Select Visual In-Context Demonstrations

    cs.LG 2026-03 reject novelty 5.0 of 10

    A Dueling-DQN agent selects visual in-context demonstrations and outperforms kNN retrieval on objective regression benchmarks but not on subjective preference tasks, per the paper's main table.

Reference graph

Works this paper leans on

76 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Aky \" u rek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=0g0X4H8yN4I

    work page 2023

  2. [2]

    How do in-context examples affect compositional generalization? In Anna Rogers, Jordan L

    Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Jian - Guang Lou, and Dongmei Zhang. How do in-context examples affect compositional generalization? In Anna Rogers, Jordan L. Boyd - Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toron...

    doi:10.18653/v1/2023.acl-long.618 2023

  3. [3]

    Skill-based few-shot selection for in-context learning

    Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, and Jian-Guang Lou. Skill-based few-shot selection for in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13472--13492, Singapore, December 2023 b . Associati...

    doi:10.18653/v1/2023.emnlp-main.831 2023

  4. [4]

    A theory for emergence of complex skills in language models

    Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936, 2023

    arXiv 2023

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  6. [6]

    Data curation alone can stabilize in-context learning

    Ting-Yun Chang and Robin Jia. Data curation alone can stabilize in-context learning. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023

    work page 2023

  7. [7]

    Fast greedy map inference for determinantal point process to improve recommendation diversity

    Laming Chen, Guoxin Zhang, and Eric Zhou. Fast greedy map inference for determinantal point process to improve recommendation diversity. Advances in Neural Information Processing Systems, 31, 2018

    work page 2018

  8. [8]

    On the relation between sensitivity and accuracy in in-context learning

    Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. On the relation between sensitivity and accuracy in in-context learning. In 2023 Findings of the Association for Computational Linguistics: EMNLP 2023, pages 155--167. Association for Computational Linguistics (ACL), 2023

    work page 2023

Show all 76 references
  1. [9]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  2. [10]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  3. [11]

    Cook, John P

    Ryan A. Cook, John P. Lalor, and Ahmed Abbasi. No simple answer to data complexity: An examination of instance-level complexity metrics for classification tasks. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Ameri...

    2025

  4. [12]

    Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4005--4019, 2023

    2023

  5. [13]

    In-context learning and gradient descent revisited

    Gilad Deutch, Nadav Magar, Tomer Natan, and Guy Dar. In-context learning and gradient descent revisited. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p...

    2024

  6. [14]

    Metacognitive capabilities of llms: An exploration in mathematical problem solving

    Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Sanjeev Arora. Metacognitive capabilities of llms: An exploration in mathematical problem solving. arXiv preprint arXiv:2405.12205, 2024

    2024 arXiv

  7. [15]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  8. [16]

    Constructing effective in-context demonstration for code intelligence tasks: An empirical study

    Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, and Michael R Lyu. Constructing effective in-context demonstration for code intelligence tasks: An empirical study. CoRR, 2023

    2023

  9. [17]

    What can transformers learn in-context? a case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35: 0 30583--30598, 2022

    2022

  10. [18]

    Dual: Diversity and uncertainty active learning for text summarization, 2025

    Petros Stylianos Giouroukis, Alexios Gidiotis, and Grigorios Tsoumakas. Dual: Diversity and uncertainty active learning for text summarization, 2025. URL https://arxiv.org/abs/2503.00867

    2025 arXiv

  11. [19]

    Coverage-based example selection for in-context learning

    Shivanshu Gupta, Matt Gardner, and Sameer Singh. Coverage-based example selection for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13924--13950, 2023

    2023

  12. [20]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  13. [21]

    A latent space theory for emergent abilities in large language models

    Hui Jiang. A latent space theory for emergent abilities in large language models. arXiv preprint arXiv:2304.09960, 2023

    2023 arXiv

  14. [22]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), page...

    2020

  15. [23]

    Diverse demonstrations improve in-context compositional generalization

    Itay Levy, Ben Bogin, and Jonathan Berant. Diverse demonstrations improve in-context compositional generalization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1401--1422, 2023

    2023

  16. [24]

    Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers

    Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of t...

    2024 doi

  17. [25]

    The closeness of in-context learning and weight shifting for softmax regression

    Shuai Li, Zhao Song, Yu Xia, Tong Yu, and Tianyi Zhou. The closeness of in-context learning and weight shifting for softmax regression. arXiv preprint arXiv:2304.13276, 2023 a

    2023 arXiv

  18. [26]

    Finding support examples for in-context learning

    Xiaonan Li and Xipeng Qiu. Finding support examples for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6219--6235, 2023 a

    2023

  19. [27]

    Finding support examples for in-context learning, 2023 b

    Xiaonan Li and Xipeng Qiu. Finding support examples for in-context learning, 2023 b . URL https://arxiv.org/abs/2302.13539

    2023 arXiv

  20. [28]

    Transformers as algorithms: Generalization and stability in in-context learning

    Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pages 19565--19594. PMLR, 2023 b

    2023

  21. [29]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  22. [30]

    What makes good in-context examples for gpt- 3 ? arXiv preprint arXiv:2101.06804, 2021

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt- 3 ? arXiv preprint arXiv:2101.06804, 2021

    2021 arXiv

  23. [31]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024

    2024

  24. [32]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786, 2021

    2021 arXiv

  25. [33]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2022

  26. [34]

    Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y Zhao. Dr. icl: Demonstration-retrieved in-context learning. arXiv preprint arXiv:2305.14128, 2023

    2023 arXiv

  27. [35]

    In-context learning with retrieved demonstrations for language models: A survey, 2024

    Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. In-context learning with retrieved demonstrations for language models: A survey, 2024. URL https://arxiv.org/abs/2401.11624

    2024 arXiv

  28. [36]

    Problem-solving logic guided curriculum in-context learning for llms complex reasoning, 2025

    Xuetao Ma, Wenbin Jiang, and Hua Huang. Problem-solving logic guided curriculum in-context learning for llms complex reasoning, 2025. URL https://arxiv.org/abs/2502.15401

    2025

  29. [37]

    Learning word vectors for sentiment analysis

    Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142--150, 2011

    2011

  30. [38]

    Hidden factors and hidden topics: understanding rating dimensions with review text

    Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165--172, 2013

    2013

  31. [39]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, page...

    2022

  32. [40]

    What in-context learning “learns” in-context: Disentangling task recognition and task learning

    Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning “learns” in-context: Disentangling task recognition and task learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8298--8319, 2023

    2023

  33. [41]

    In-context learning with iterative demonstration selection

    Chengwei Qin, Aston Zhang, Chen Chen, Anirudh Dagar, and Wenming Ye. In-context learning with iterative demonstration selection. arXiv preprint arXiv:2310.09881, 2023

    2023 arXiv

  34. [42]

    SQ u AD : 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQ u AD : 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages...

    2016 doi

  35. [43]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  36. [44]

    Learning to retrieve prompts for in-context learning

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655--2671, 2022

    2022

  37. [45]

    Reticl: Sequential retrieval of in-context examples with reinforcement learning

    Alexander Scarlatos and Andrew Lan. Reticl: Sequential retrieval of in-context examples with reinforcement learning. arXiv preprint arXiv:2305.14502, 2023

    2023 arXiv

  38. [46]

    Active learning for convolutional neural networks: A core-set approach, 2018

    Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach, 2018. URL https://arxiv.org/abs/1708.00489

    2018 arXiv

  39. [47]

    Do pretrained transformers really learn in-context by gradient descent? arXiv preprint arXiv:2310.08540, 2023

    Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Do pretrained transformers really learn in-context by gradient descent? arXiv preprint arXiv:2310.08540, 2023

    2023 arXiv

  40. [48]

    Diversifying convex transductive experimental design for active learning

    Lei Shi and Yi-Dong Shen. Diversifying convex transductive experimental design for active learning. In IJCAI, pages 1997--2003, 2016

    1997

  41. [49]

    Smith, and Tao Yu

    Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Represe...

    2023

  42. [50]

    Smith, and Yejin Choi

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the ...

    2020 doi

  43. [51]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

    2019

  44. [52]

    Using multiple clause constructors in inductive logic programming for semantic parsing

    Lappoon R Tang and Raymond J Mooney. Using multiple clause constructors in inductive logic programming for semantic parsing. In European Conference on Machine Learning, pages 466--477. Springer, 2001

    2001

  45. [53]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  46. [54]

    Transformers learn in-context by gradient descent

    Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo \ a o Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151--35174. PMLR, 2023 a

    2023

  47. [55]

    Uncovering mesa-optimization algorithms in transformers

    Johannes Von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklasson, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.05858, 2023 b

    2023 arXiv

  48. [56]

    Contributing dimension structure of deep feature for coreset selection, 2024

    Zhijing Wan, Zhixiang Wang, Yuran Wang, Zheng Wang, Hongyuan Zhu, and Shin'ichi Satoh. Contributing dimension structure of deep feature for coreset selection, 2024. URL https://arxiv.org/abs/2401.16193

    2024 arXiv

  49. [57]

    Diversity measurement and subset selection for instruction tuning datasets, 2024

    Peiqi Wang, Yikang Shen, Zhen Guo, Matthew Stallone, Yoon Kim, Polina Golland, and Rameswar Panda. Diversity measurement and subset selection for instruction tuning datasets, 2024. URL https://arxiv.org/abs/2402.02318

    2024 arXiv

  50. [58]

    Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning

    Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. In Workshop on Efficient Systems for Foundation Models@ ICML2023

  51. [59]

    Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning

    Xinyi Wang, Wanrong Zhu, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint arXiv:2301.11916, page 3, 2023

    2023 arXiv

  52. [60]

    Crowdsourcing multiple choice science questions

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94--106, 2017

    2017

  53. [61]

    The learnability of in-context learning

    Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. Advances in Neural Information Processing Systems, 36: 0 36637--36651, 2023

    2023

  54. [62]

    Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering

    Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volum...

    2023

  55. [63]

    An explanation of in-context learning as implicit bayesian inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL htt...

    2022

  56. [64]

    Representative demonstration selection for in-context learning with two-stage determinantal point process

    Zhao Yang, Yuanzhe Zhang, Dianbo Sui, Cao Liu, Jun Zhao, and Kang Liu. Representative demonstration selection for in-context learning with two-stage determinantal point process. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 54...

    2023

  57. [65]

    An empirical study of GPT-3 for few-shot knowledge-based VQA

    Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of GPT-3 for few-shot knowledge-based VQA . In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications...

    2022 doi

  58. [66]

    Complementary explanations for effective in-context learning

    Xi Ye, Srinivasan Iyer, Asli Celikyilmaz, Ves Stoyanov, Greg Durrett, and Ramakanth Pasunuru. Complementary explanations for effective in-context learning. Findings of the Association for Computational Linguistics: ACL 2023, 2023

    2023

  59. [67]

    Ground-truth labels matter: A deeper look into input-label demonstrations

    Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. Ground-truth labels matter: A deeper look into input-label demonstrations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,...

    2022

  60. [68]

    SKILL-MIX: a flexible and expandable family of evaluations for AI models

    Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown - Cohen, Anirudh Goyal, and Sanjeev Arora. SKILL-MIX: a flexible and expandable family of evaluations for AI models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 202...

    2024

  61. [69]

    Enhancing sample selection against label noise by cutting mislabeled easy examples, 2025

    Suqin Yuan, Lei Feng, Bo Han, and Tongliang Liu. Enhancing sample selection against label noise by cutting mislabeled easy examples, 2025. URL https://arxiv.org/abs/2502.08227

    2025

  62. [70]

    Learning to parse database queries using inductive logic programming

    John M Zelle and Raymond J Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pages 1050--1055, 1996

    1996

  63. [71]

    Toso, and James Anderson

    Donglin Zhan, Leonardo F. Toso, and James Anderson. Coreset-based task selection for sample-efficient meta-reinforcement learning, 2025 a . URL https://arxiv.org/abs/2502.02332

    2025 arXiv

  64. [72]

    Mmrag: Multi-mode retrieval-augmented generation with large language models for biomedical in-context learning, 2025 b

    Zaifu Zhan, Jun Wang, Shuang Zhou, Jiawen Deng, and Rui Zhang. Mmrag: Multi-mode retrieval-augmented generation with large language models for biomedical in-context learning, 2025 b . URL https://arxiv.org/abs/2502.15954

    2025 arXiv

  65. [73]

    What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization

    Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305.19420, 2023 a

    2023 arXiv

  66. [74]

    Automatic chain of thought prompting in large language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023 b . URL https://openreview....

    2023

  67. [75]

    Can models learn skill composition from examples? Advances in Neural Information Processing Systems, 37: 0 102393--102427, 2025

    Haoyu Zhao, Simran Kaur, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Can models learn skill composition from examples? Advances in Neural Information Processing Systems, 37: 0 102393--102427, 2025

    2025

  68. [76]

    Calibrate before use: Improving few-shot performance of language models

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697--12706. PMLR, 2021

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.