Pith. sign in

REVIEW 4 major objections 6 minor 43 references

Towards Compute-Optimal Many-Shot In-Context Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a many-shot prompt can reach similarity-based accuracy while staying almost fully cacheable, by mixing a small per-test similar set with a large fixed demonstration pool.

desk verdict A practical, clearly-written hybrid selection recipe for many-shot ICL that mostly delivers on its cost/accuracy promise, though the evidence would be stronger with error bars, an s-ablation, and numeric tables. read the letter →

arxiv 2507.16217 v2 pith:VOUIEPHN submitted 2025-07-22 cs.CL cs.AIcs.LG

Shahriar Golchin , Yanfei Chen , Rujun Han , Manan Gandhi , Tianli Yu , Swaroop Mishra , Mihai Surdeanu , Rishabh Agarwal
show 2 more authors
Chen-Yu Lee Tomas Pfister
This is my paper
classification cs.CLcs.AIcs.LG
keywords many-shotin-contextlearningdemonstrationselectionpromptcachingsimilarity-basedk-meansclusteringinferencecostlong-contextlanguagemodelshybridprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether many-shot in-context learning can keep the accuracy gains of similarity-based demonstration selection without paying its quadratic inference cost. It answers by splitting the prompt: a small set of demonstrations similar to the test input is computed fresh for each query, while the much larger remainder is chosen randomly or by k-means clustering and cached across all queries. Experiments on Gemini Pro and Flash over ANLI, TREC, GSM Plus, and MetaTool show that the hybrid prompts consistently beat random selection and match or exceed the strongest similarity-based baseline, with estimated inference cost reduced by roughly a factor of two at 50 shots and up to an order of magnitude at 200 shots. The paper also shows that the share of similar demonstrations can be tuned to trade accuracy against cost, and that in a low-data regime the hybrid still beats using every available demonstration.

What carries the argument

The central object is the hybrid many-shot prompt, in which a constant $s=20$ demonstrations selected by cosine similarity to the test embedding are added to a large fixed, cacheable block. The fixed block is either $r$ random demonstrations or $k$ demonstrations chosen by k-means centroid mapping: test-sample embeddings are clustered, the optimal number of clusters is set by the elbow method, and for each centroid the nearest demonstrations in the pool are cached. Attention-complexity accounting is the argument's engine: with key-value caching the per-query cost is $O(c(s+t))$ with cached token count $c$ dominating the similar demonstrations $s$ and the test tokens $t$, hence linear in $c$, whereas full per-query similarity selection costs $O(n^2)$. The diminishing-effectiveness premise justifies why $r$ or $k$ can greatly exceed $s$ without hurting accuracy.

What would settle it

Run a 200-shot comparison on a task where accuracy under full similarity-based selection still rises sharply between 100 and 200 shots; the hybrid predicts parity, so any large gap would show the cached majority is not neutral and would break the method.

Watch

Extended reading notes

Core claim

The paper's central discovery is that selection criteria have diminishing marginal value in many-shot in-context learning: after roughly 50 to 100 demonstrations, the few most similar demonstrations carry most of the benefit that similarity selection provides, so the rest can be chosen cheaply and cached. Concretely, an $n$-shot prompt is built as $s=20$ test-specific similar demonstrations plus $r$ (or $k$) cached random (or k-means centroid-selected) demonstrations with $r,k \gg s$. The authors argue that this hybrid is compute-optimal in the sense that it reaches the accuracy of fully dynamic similarity-based selection while preserving the $O(c)$ inference scaling of fully cached methods, instead of the $O(n^2)$ scaling of similarity selection. On their evaluated datasets, the hybrid matches or outperforms the strongest baseline, and the cached majority adds a compound benefit rather than diluting the similar demonstrations.

Load-bearing premise

The method assumes that when a prompt already contains a hundred or more demonstrations, the extra accuracy from choosing examples most similar to the test question is small, so most of the prompt can be filled with cheap cached examples without hurting performance.

Editorial extensions

If this is right

  • Practitioners can adopt a largely cached many-shot prompt while still customizing it for each test sample, removing the main obstacle to scaling demonstrations.
  • At 200 shots the estimated inference cost is around ten times lower than full similarity-based selection, with performance at parity or better on the evaluated tasks.
  • The ratio of similar to cached demonstrations becomes a tunable hyperparameter that trades accuracy against inference cost in many-shot ICL.
  • Even with only a 100-example demonstration pool, the hybrid strategies outperform using the full pool, so the approach does not require a large curated set.
  • Because only the similar subset is recomputed, the method composes with any caching implementation, including key-value cache.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same cost argument should extend beyond 200 demonstrations: as long as the diminishing-effectiveness premise holds, the hybrid structure keeps inference linear while the cached pool grows, so the method could plausibly scale to thousands of demonstrations.
  • The premise implies a testable boundary: tasks where later similar demonstrations still add accuracy, such as highly heterogeneous or label-dense tasks, may need a larger $s$, and adapting $s$ per task could be a practical follow-up the paper does not pursue.
  • An implicit economic consequence is that the expensive part of computation shifts to offline embedding and clustering steps, which are amortized; whether that is worthwhile depends on query volume and context lengths, a comparison the paper does not quantify.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes two hybrid demonstration-selection strategies for many-shot in-context learning. The first combines a small number (s=20) of test-similar demonstrations with a much larger cached random set; the second replaces the random cached set with demonstrations chosen via k-means centroids computed from test-sample representations. The central claim is that these hybrids match or surpass fully similarity-based selection while keeping inference cost near that of fully cached methods, reducing the estimated cost by up to an order of magnitude at 200 shots. The paper evaluates the methods on Gemini Pro and Flash across ANLI, TREC, GSM Plus, and MetaTool, plus low-data BBH subsets, and reports performance-cost Pareto results. The motivation rests on a 'diminishing effectiveness of selection criteria' premise, which the authors say they verify in their own experiments.

Significance. If the central claim holds, the paper provides a simple and practical recipe for many-shot ICL that preserves most of the accuracy of dynamic similarity-based selection while remaining largely cacheable. The cost model is an analytic complexity bound rather than a fitted curve, which is a strength, and the paper includes two k-means baselines to isolate the effect of centroid mapping from test data. The proposed methods are model-agnostic and cache-agnostic. However, the empirical support is currently limited by the absence of repeated seeds, error bars, and numeric tables, and by the lack of an ablation on the key hyperparameter s=20. The significance of the contributions is therefore conditional on these being addressed.

major comments (4)
  1. [Section 3 (Experimental Setup) and Section 4.4 (Figure 5)] The main experiments fix s=20 similar demonstrations for all prompt sizes, and no ablation over s is reported for the primary four-dataset setup. The paper's own low-data results in Figure 5 show that increasing the similar share from 20 to 80 improves accuracy by about 3–6%, which indicates that s=20 is not universally sufficient to capture the similarity benefit. Because the central parity claim depends on the premise that similarity's marginal benefit beyond 20 examples is negligible, the absence of an s-ablation leaves the headline claim unsupported. Please add an ablation over s at fixed total n for at least a subset of the datasets and models, and provide a quantitative measure of diminishing returns as a function of the number of similar demonstrations.
  2. [Section 4.2 and Figure 4] The paper's own analysis of TREC states that random selection continues to improve up to 200 shots and has not yet approximated similarity-based selection by 200 shots. This directly weakens the diminishing-effectiveness premise for the random component within the tested range. Since the hybrid methods replace up to 180 of 200 demonstrations with cached random or k-means demonstrations, a task with a longer-tailed selection benefit could cause the cached majority to drag accuracy below the full-similarity baseline. The verification of the premise in Section 4.2 is qualitative (reading trends off Figure 4) rather than quantitative. Please report the marginal accuracy gain of similarity-based selection as a function of the number of similar demonstrations, rather than only visual inspection.
  3. [Section 4 and Figure 4] The paper reports no error bars, no repeated seeds, and no statistical significance tests, and it does not provide numeric tables of the plotted values. Claims such as 'consistently outperform random selection and surpass or match the most performant selection approach' in the abstract cannot be assessed for reliability without variance estimates. Please provide mean and standard deviation over at least three to five seeds (or an equivalent bootstrap) for the key configurations, and include the numerical values underlying Figures 2, 4, and 5 in an appendix or supplementary table.
  4. [Section 4.1 (Inference Cost Analysis)] The 'up to an order of magnitude' inference-cost reduction is derived from an asymptotic attention-complexity model, O(c(s+t)) versus O(n^2), and is not validated with any measured inference time. Since the paper's title and framing emphasize compute-optimality, the cost claim should either be supported by end-to-end latency measurements on the evaluated models or explicitly presented as a model-based estimate with the simplifying assumptions stated, including the treatment of output generation, prefill, and memory-bandwidth effects.
minor comments (6)
  1. [Section 2.3] The sentence 'This method aims to enhance our first strategy byencouraging diversity in demonstration selection' contains a typo ('byencouraging') and should read 'by encouraging'.
  2. [Section 4.3] The quantitative claims 'nearly two times lower in 50-shot settings and ten times lower in 200-shot settings' are not tied to a specific formula or table; please derive them explicitly from the cost model in Section 4.1 or cite the relevant panel in Figure 4.
  3. [Section 4.4] The low-data experiment varies the number of similar demonstrations and the number of cached demonstrations simultaneously while holding the total at 100, so the observed 3–6% gains conflate the effect of adding more similar demonstrations with the effect of removing cached ones. This should be acknowledged, and an additional ablation that varies s while keeping the cached-set size fixed would isolate the effect of s.
  4. [Figure 4] The panels do not include error bars or numeric labels for all data points, making it difficult to recover exact values; including a table of the plotted numbers in an appendix would improve reproducibility.
  5. [Section 3 (Models)] The paper states that temperature was set to zero 'to promote deterministic results,' but decoding can still be nondeterministic in some serving setups; please clarify whether greedy decoding was used and whether any runs were repeated to confirm determinism.
  6. [Section 2.1 and References] The diminishing-effectiveness premise is attributed to Agarwal et al. (2024) and Bertsch et al. (2024), but no specific sections, figures, or tables of those works are cited; pointing to the relevant evidence would make the premise more reproducible.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper's headline claims are empirical measurements plus an analytic cost model, with no fitted parameter relabeled as a prediction.

full rationale

The paper's central claims are empirical measurements of accuracy across selection strategies, not derived quantities. The hybrid method is constructed by combining s similar demonstrations with a cached random or k-means set, but the claim that it matches or surpasses similarity-based selection is tested against baselines on held-out test samples, not implied by the construction. The inference-cost analysis is an analytic complexity bound with no fitted parameters and is not a renamed empirical result; it follows from attention complexity and the assumption that cached content is reused. The 'diminishing effectiveness of selection criteria' premise is cited from Agarwal et al. (2024) and Bertsch et al. (2024), and the paper explicitly verifies it in Section 4.2 using the same experimental setup, so it is not assumed by construction. The only self-citation in this load-bearing motivation (Agarwal et al. 2024, where Rishabh Agarwal is a co-author of the present paper) is corroborated by an external citation and by the paper's own experiments, so it does not constitute circular reasoning. No fitted parameter is relabeled as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed. The paper is self-contained as an empirical study, and any concerns about the sufficiency of s=20 or test-data use in choosing k-means cluster counts are correctness or robustness issues, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on one hand-chosen hyperparameter (s=20), data-dependent cluster counts, and domain assumptions about caching and selection-criterion effectiveness. No new entities are introduced. The most consequential premise is the diminishing-effectiveness assumption, which the paper verifies only on the tested models and datasets.

free parameters (3)
  • s = number of similar demonstrations = 20
    Fixed at 20 across all main experiments (Section 3). This hyperparameter controls the balance between performance and cost and is chosen by hand; no sensitivity study is provided in the main setting.
  • number of k-means clusters = 1 to 20, selected per dataset via elbow method
    The number of centroids is selected per dataset using inertia and the elbow method (Section 3, k-Means Clustering), making the cached demonstration set data-dependent.
  • r/k values for cached demonstrations = 0, 30, 80, 130, 180
    Together with s=20, these define the n-shot settings; the ratio is treated as a tunable hyperparameter and the paper shows it balances performance and cost, but does not prescribe a selection rule.
assumptions (5)
  • standard math Transformer attention cost is O(n^2) for uncached prompts and O(ct + t^2) with KV caching.
    Used in Section 4.1 to derive proportional inference costs.
  • domain assumption In many-shot ICL, similarity-based selection loses its advantage over random selection as the number of demonstrations grows, so a large cached random or k-means set is nearly as good as a similarity-selected set.
    Core premise of the hybrid design, introduced in Section 2.1 and verified only on the tested datasets in Section 4.2; if false, the hybrid methods would not match similarity-based accuracy.
  • domain assumption Gecko text embeddings with cosine similarity are a valid proxy for demonstration usefulness on these tasks.
    Similarity and k-means selection both rely on embedding representations (Section 3); no independent validation of this proxy is provided.
  • ad hoc to paper Test-sample representations may be used to choose cached demonstrations.
    The k-means variant and k-means-with-mapping baseline compute centroids from the test set (Section 2.3), giving those methods transductive access that the random baseline does not have.
  • domain assumption KV caching of demonstrations requires no recomputation and introduces no approximation error.
    The cost model in Section 4.1 assumes cached tokens are attended to at O(ct) cost with no quality loss; actual cache behavior and overhead are not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Compute-Optimal Many-Shot In-Context Learning." pith.science (2026). https://pith.science/paper/VOUIEPHN

@misc{pith2026250716217,
  author       = {Pith},
  title        = {Pith review of: Towards Compute-Optimal Many-Shot In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VOUIEPHN}},
  note         = {Machine review of arXiv:2507.16217}
}
read the original abstract

Long-context large language models (LLMs) are able to process inputs containing up to several million tokens. In the scope of in-context learning (ICL), this translates into using hundreds/thousands of demonstrations in the input prompt, enabling many-shot ICL. In practice, a fixed set of demonstrations is often selected at random in many-shot settings due to (1) high inference costs, (2) the benefits of caching and reusing computations, and (3) the similar performance offered by this strategy compared to others when scaled. In this work, we propose two straightforward strategies for demonstration selection in many-shot ICL that improve performance with minimal computational overhead. Our first method combines a small number of demonstrations, selected based on their similarity to each test sample, with a disproportionately larger set of random demonstrations that are cached. The second strategy improves the first by replacing random demonstrations with those selected using centroids derived from test sample representations via k-means clustering. Our experiments with Gemini Pro and Flash across several datasets indicate that our strategies consistently outperform random selection and surpass or match the most performant selection approach while supporting caching and reducing inference cost by up to an order of magnitude. We also show that adjusting the proportion of demonstrations selected based on different criteria can balance performance and inference cost in many-shot ICL.

Figures

Figures reproduced from arXiv: 2507.16217 by the authors.

Figure 2
Figure 2. Pareto analysis of performance vs. inference cost across various datasets, compar [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 1
Figure 1. Our proposed many-shot ICL prompt format. Hatched blocks represent cached content, while solid blocks indicate non-cached content. When LLM is prompted, new com￾putations are performed only for a small set of demonstrations selected based on similarity to the down￾stream test sample, while previous computations for a larger set of ran￾dom or k-means-selected demonstra￾tions are reused. Block sizes in the figure aim … view at source ↗
Figure 3
Figure 3. Illustration of selecting demonstra￾tions using k-means clustering. First, centroids are computed based on the representations of test samples using the elbow method. These cen￾troids are then mapped to the representations of available demonstrations. Lastly, demonstra￾tions that are most similar to each centroid are selected based on a predefined total. In this example, the predefined total is 15 demonstra￾tions, w… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Results from Gemini Pro and Flash on various datasets in many-shot ICL settings. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Results across various subsets of the BBH dataset in a 100-shot ICL setting under a [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: An example of the fixed, cached part of the input prompt for the ANLI dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: An example of the dynamic, uncached part of the input prompt for the ANLI [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: An example of the fixed, cached part of the input prompt for the TREC dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: An example of the dynamic, uncached part of the input prompt for the TREC [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: An example of the fixed, cached part of the input prompt for the GSM Plus [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: An example of the dynamic, uncached part of the input prompt for the GSM Plus [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: An example of the fixed, cached part of the input prompt for the MetaTool dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: An example of the dynamic, uncached part of the input prompt for the MetaTool [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: An example of the fixed, cached part of the input prompt for the BBH dataset, [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: An example of the dynamic, uncached part of the input prompt for the BBH [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: An example of the fixed, cached part of the input prompt for the BBH dataset, [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: An example of the dynamic, uncached part of the input prompt for the BBH [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: An example of the fixed, cached part of the input prompt for the BBH dataset, [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: An example of the dynamic, uncached part of the input prompt for the BBH [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 16 canonical work pages

  1. [1]

    Many-shot in-context learning

    Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018,

    arXiv

  2. [3]

    Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al

    URL https://openreview.net/forum?id=cw5mgd71jW. Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,

    arXiv

  3. [5]

    In-context learning with long-context models: An in-depth exploration

    Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200,

    arXiv

  4. [6]

    impact of sample selection on in-context learning for entity extraction from scientific writing

    Necva B¨ol ¨uc ¨u, Maciej Rybinski, and Stephen Wan. impact of sample selection on in-context learning for entity extraction from scientific writing. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5090–5107, Singapore, December

    work page 2023

  5. [7]

    doi: 10.18653/v1/2023.findings-emnlp.338

    Association for Computational Linguis- tics. doi: 10.18653/v1/2023.findings-emnlp.338. URL https://aclanthology.org/2023. findings-emnlp.338/. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in ne...

    work page doi:10.18653/v1/2023.findings-emnlp.338 2023

  6. [9]

    Papailiopoulos, and Kangwook Lee

    Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris S. Papailiopoulos, and Kangwook Lee. LIFT: language- interfaced fine-tuning for non-language machine learning tasks. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 3...

    work page 2022

  7. [11]

    Data engineering for scaling language models to 128k context

    10 Published as a conference paper at COLM 2025 Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10171,

    arXiv 2025

  8. [12]

    Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context

    Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530,

    arXiv

Show all 43 references
  1. [13]

    Time travel in llms: Tracing data contamina- tion in large language models

    Shahriar Golchin and Mihai Surdeanu. Time travel in llms: Tracing data contamina- tion in large language models. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net,

    2024

  2. [14]

    Shahriar Golchin, Mihai Surdeanu, Steven Bethard, Eduardo Blanco, and Ellen Riloff

    URL https://openreview.net/forum?id=2Rwq6c3tvr. Shahriar Golchin, Mihai Surdeanu, Steven Bethard, Eduardo Blanco, and Ellen Riloff. Memorization in in-context learning. arXiv preprint arXiv:2408.11546,

    arXiv

  3. [16]

    Accessed: 2025-03-28

    URL https:// cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings . Accessed: 2025-03-28. Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples. arXiv preprint arXiv:2212.06713,

    2025 arXiv

  4. [17]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. arXiv preprint arXiv:2310.15916,

    arXiv

  5. [19]

    Many-shot in-context learning in multimodal foundation models

    Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y Ng. Many-shot in-context learning in multimodal foundation models. arXiv preprint arXiv:2405.09798,

    arXiv

  6. [20]

    In-context learning in large language models learns label relationships but is not conventional learning

    Jannik Kossen, Tom Rainforth, and Yarin Gal. In-context learning in large language models learns label relationships but is not conventional learning. arXiv preprint arXiv:2307.12375,

    arXiv

  7. [21]

    Gecko: Versatile text embeddings distilled from large language models

    Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327,

    arXiv

  8. [22]

    Llms-as-judges: A comprehensive survey on llm-based evaluation methods

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Llms-as-judges: A comprehensive survey on llm-based evaluation methods. CoRR, abs/2412.05579, 2024a. doi: 10.48550/ARXIV .2412.05579. URL https://doi.org/10. 48550/arXiv.2412.05579. Q...

    doi arXiv

  9. [24]

    What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804,

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804,

    arXiv

  10. [26]

    Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837,

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837,

    arXiv

  11. [27]

    Many-shot in-context learning for molecular inverse design

    Saeed Moayedpour, Alejandro Corrochano-Navarro, Faryad Sahneh, Shahriar Norooz- izadeh, Alexander Koetter, Jiri Vymetal, Lorenzo Kogler-Anele, Pablo Mas, Yasser Jangjou, Sizhen Li, et al. Many-shot in-context learning for molecular inverse design. arXiv preprint arXiv:2407.19089,

    arXiv

  12. [28]

    Adversarial nli: A new benchmark for natural language understanding

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599,

    1910 arXiv

  13. [29]

    Can many-shot in-context learning help llms as evaluators? A preliminary empirical study

    Mingyang Song, Mao Zheng, and Xuan Luo. Can many-shot in-context learning help llms as evaluators? A preliminary empirical study. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st Interna...

    2025

  14. [30]

    Mirac Suzgun, Nathan Scales, Nathanael Sch¨arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al

    URL https://aclanthology.org/2025.coling-main.548/. Mirac Suzgun, Nathan Scales, Nathanael Sch¨arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve th...

    2025 arXiv

  15. [32]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett ...

    2017

  16. [34]

    press/v202/von-oswald23a.html

    URL https://proceedings.mlr. press/v202/von-oswald23a.html. Xingchen Wan, Ruoxi Sun, Hootan Nakhost, and Sercan O Arik. Teach better or show smarter? on instructions and exemplars in automatic prompt optimization. arXiv preprint arXiv:2406.15708,

    arXiv

  17. [35]

    From few to many: Self-improving many-shot reasoners through iterative optimization and generation

    Xingchen Wan, Han Zhou, Ruoxi Sun, Hootan Nakhost, Ke Jiang, and Sercan ¨O Arık. From few to many: Self-improving many-shot reasoners through iterative optimization and generation. arXiv preprint arXiv:2502.00330,

    arXiv

  18. [36]

    Ground-truth labels matter: A deeper look into input-label demonstrations

    Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. Ground-truth labels matter: A deeper look into input-label demonstrations. arXiv preprint arXiv:2205.12685,

    arXiv

  19. [37]

    Sentiment analysis in the era of large language models: A reality check

    Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. Sentiment analysis in the era of large language models: A reality check. arXiv preprint arXiv:2305.15005,

    arXiv

  20. [38]

    Active example selection for in-context learning

    Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. arXiv preprint arXiv:2211.04486,

    arXiv

  21. [39]

    Calibrate before use: Improving few-shot performance of language models

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Vi...

    2021

  22. [40]

    In particular, green highlights the instruction, blue represents demonstrations selected ei- ther randomly or using k-means clustering, red indicates similar demonstrations that are dynamically included in the input prompt for each downstream test sample, and purple shows the ...

    2025

  23. [42]

    This portion of the prompt is updated for each downstream test sample

    How many years ago was Raymond’s son born? Solution: Figure 11: An example of the dynamic, uncached part of the input prompt for the GSM Plus dataset. This portion of the prompt is updated for each downstream test sample. 18 Published as a conference paper at COLM 2025 Instruc...

    2025

  24. [1953]

    Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, and Mihai Surdeanu

    doi: 10.1007/BF02289263. Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, and Mihai Surdeanu. From words to numbers: Your large language model is secretly a capable regressor when given in-context examples. arXiv preprint arXiv:2404.07544,

    arXiv

  25. [1991]

    Hypothesis: Oil prices will be affected by US elections. Label: Neutral – – – [...] other {Random Demonstrations } or {k-Means-Selected Demonstrations } – – – Figure 6: An example of the fixed, cached part of the input prompt for the ANLI dataset. This prompt is followed by dy...

    2025

  26. [2001]

    Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun

    URL https://www.aclweb.org/ anthology/H01-1069. Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. Metatool benchmark: Deciding whether to use tools and which to use. arXiv preprint arXiv: 231...

    arXiv

  27. [2002]

    11 Published as a conference paper at COLM 2025 Ziqian Lin and Kangwook Lee

    URL https://www.aclweb.org/anthology/ C02-1150. 11 Published as a conference paper at COLM 2025 Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning. arXiv preprint arXiv:2402.18819,

    2025 arXiv

  28. [2015]

    That error will be one of the following types: Named Entities: An entity (names, places, locations, etc.) is changed to a different entity

    The translation contains an error pertaining to (A) Modifiers or Adjectives (B) Numerical Values (C) Negation or Antonyms (D) Named Entities (E) Dropped Content (F) Facts Target: B – – – Input: The following translations from German to English contain a particular error. That ...

    2025

  29. [2017]

    Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo ˜ao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov

    URLhttps://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html. Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo ˜ao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient ...

    2017

  30. [2020]

    Longrope: Extending llm context window beyond 2 million tokens

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753,

    arXiv

  31. [2021]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786,

    arXiv

  32. [2022]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al

    URL http://papers.nips.cc/paper files/paper/2022/hash/ 4ce7fe1d2730f53cb3857032952cd1b8-Abstract-Conference.html. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv prep...

    2022 arXiv

  33. [2023]

    Revisit- ing in-context learning with long context language models.arXiv preprint arXiv:2412.16926,

    Jinheon Baek, Sun Jae Lee, Prakhar Gupta, Siddharth Dalmia, Prateek Kolhar, et al. Revisit- ing in-context learning with long context language models.arXiv preprint arXiv:2412.16926,

    arXiv

  34. [2024]

    How do in-context examples affect compositional generalization? arXiv preprint arXiv:2305.04835,

    Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Jian-Guang Lou, and Dongmei Zhang. How do in-context examples affect compositional generalization? arXiv preprint arXiv:2305.04835,

    arXiv

  35. [2025]

    Accessed: 2025-02-13

    URL https://ai.google.dev/ gemini-api/docs/caching. Accessed: 2025-02-13. Google Cloud. Get text embeddings with generative ai. https://cloud.google.com/ vertex-ai/generative-ai/docs/embeddings/get-text-embeddings ,

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.