Pith. sign in

REVIEW 5 major objections 6 minor 64 references

Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusions

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LCS, a slope between learning gain and contextual relevance, is proposed as a reliable measure of in-context learning effectiveness, with a 0.2 threshold for label-free diagnosis.

desk verdict The LCS slope is a useful labeled-data diagnostic, but the label-free extension rests on a false theorem and should not be trusted as presented. read the letter →

arxiv 2506.23146 v3 pith:R2JXVGNP submitted 2025-06-29 cs.CL

Dingzriui Wang , Xuanliang Zhang , Keyan Xu , Qingfu Zhu , Wanxiang Che , Yang Deng This is my paper
classification cs.CL
keywords in-contextlearningLLMevaluationlearning-to-contextslopemetricsyntheticdatalossdecreasecontextualrelevancedemonstrationselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Learning-to-Context Slope (LCS), a way to measure how much a large language model actually learns from in-context demonstrations rather than merely whether its final answer happens to change. LCS fits a least-squares line between the loss decrease from a demonstration and the demonstration's relevance to the question; a steep slope means the model learns more when given more relevant context. Across three model families and eight datasets, the paper reports that LCS correlates strongly with performance improvement (Pearson 0.737), and that a slope near or below 0.2 marks cases where demonstrations give little or negative benefit. The paper argues this gives practitioners a label-free diagnostic: because synthetic demonstrations yield a lower-bounded LCS, even without labeled data one can decide whether labeling effort is worthwhile. A careful reader would care because performance-based checks are noisy, hard to attribute, and useless when labels are scarce.

What carries the argument

The central object is the Learning-to-Context Slope (LCS), the least-squares slope $r_{\hat p}$ fitted to pairs $(s_i, t_i)$, where $s_i$ is contextual relevance and $t_i$ is learning gain for a given example $i$. Its work is to quantify how much the model's loss decreases as the demonstration grows more relevant; the paper's key identity, Theorem 2.1, states that the two information terms are proportional with slope $p(D|Q)/p(X|Q)$, and the empirical fit estimates this relationship under sampling noise. To compute the terms, the paper first paraphrases human reasoning into the model's preferred format and then takes length-normalized token likelihoods, so that format artifacts do not inflate the measured information gain. The label-free version additionally relies on Theorem 2.2, which is used to argue that synthetic demonstrations give a lower bound on the real-data LCS.

What would settle it

Compute a counterexample to Theorem 2.2: choose a finite set of answers and demonstrations where $\hat p(X|Q;\hat D)\leq \hat p(X|Q;D^*)$ for every $X$ and $Q$, and check numerically whether $\hat p(\hat D|Q)/\hat p(\hat X|Q)\leq \hat p(D^*|Q)/\hat p(X^*|Q)$ always holds; one failure would refute the synthetic-data lower-bound claim. Alternatively, compare synthetic and real LCS on the same model-task pairs and count how often synthetic LCS above 0.2 coincides with positive performance gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that LCS, the least-squares slope of learning gain against contextual relevance, is a reliable measure of in-context learning effectiveness. For each example, learning gain is the increase in the model's probability of the demonstration when the answer is supplied, $I_p(X\to D|Q)=\hat p(D|Q;X)-\hat p(D|Q)$, and contextual relevance is the corresponding increase in the probability of the answer when the demonstration is supplied, $I_p(D\to X|Q)=\hat p(X|Q;D)-\hat p(X|Q)$. The theoretical anchor is an exact identity (Theorem 2.1) showing these two information terms are proportional, so the fitted slope is positive when demonstrations help. Experiments on Llama2-7b, Llama3.1-8b, and DeepSeek-R1-Distill-Llama-8b across math, code, reasoning, and domain-specific tasks support the claim that this slope tracks true ICL effectiveness even when accuracy does not change, and an empirical threshold of 0.2 separates effective from ineffective in-context learning. The label-free application rests on Theorem 2.2, which the paper uses to claim that LCS computed from synthetic questions is consistently lower than LCS computed from real data.

Load-bearing premise

The paper's label-free use of LCS rests on the claim that if real demonstrations always make every candidate answer more likely than synthetic demonstrations do, then the ratio of demonstration-to-answer probabilities is also larger for real data, an implication whose proof is the part most worth checking.

Editorial extensions

If this is right

  • Practitioners can measure LCS on a small set of demonstrations and learn whether in-context learning is helping before committing to a larger evaluation.
  • A threshold near 0.2 separates tasks and models where 1-shot ICL improves accuracy from those where it is flat or harmful, flagging trouble without waiting for noisy accuracy deltas.
  • Because synthetic questions under-estimate LCS, a synthetic LCS above 0.2 is a green light to invest in labeling; below-threshold synthetic scores mean the real-data LCS will be even lower.
  • ICL failures can be attributed to weak contextual alignment (the model does not connect demonstrations to the question) or strong output calibration (the model already judges the answer without demonstrations), pointing to different remedies.
  • Demonstration selection by learning gain, rather than by similarity retrieval, outperforms BM25, GTR, and IDS baselines on the tested tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to use LCS as a per-task gate in retrieval-augmented pipelines, deciding whether retrieved demonstrations are likely to help before they are inserted into the prompt.
  • The 0.2 threshold was estimated on three model families and eight datasets; a broader calibration study across more model scales, task families, and many-shot settings would show whether it is a general constant or a family-dependent heuristic.
  • The intercept of the fitted line also carries signal, falling as model capability rises, so a two-parameter diagnostic combining slope and intercept might separate 'learns from context' from 'already knows the answer' more cleanly than slope alone.
  • If LCS is truly shot-number invariant as reported, it could be estimated from a single demonstration per question, which would make the metric cheap enough to run repeatedly during prompt development.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes LCS, a least-squares slope between learning gain (defined as p(D|Q;X) - p(D|Q)) and contextual relevance (defined as p(X|Q;D) - p(X|Q)), as a metric for how effectively a model learns from in-context demonstrations. The authors derive Theorem 2.1 relating the two quantities, then use Theorem 2.2 to argue that synthetic demonstrations yield a smaller LCS than real demonstrations, which motivates a label-free threshold of 0.2. Experiments across eight datasets and three LLMs report a Pearson correlation of 0.737 between LCS and performance change, show that LCS can be informative when performance does not improve, and include an application to demonstration selection.

Significance. If the metric were fully validated, it would provide a useful, continuous, and attribution-oriented diagnostic for ICL effectiveness, with a plausible path to label-free use. The paper includes broad experiments (Tables 1, 8, 9), an application to demonstration selection (Table 2), and a clearly stated operational recipe. The authors also acknowledge scope limitations in Appendix A.1. However, the theoretical foundation for the label-free claim is not valid as stated, the threshold is selected post hoc from the same data used to report the correlation, and the metric formula in Eq. (3) appears to compute the transposed regression. These issues substantially reduce the strength of the central claims in the current version.

major comments (5)
  1. [Eq. (3), §2.2] As written, Eq. (3) computes the least-squares slope of s_i on t_i, i.e., of contextual relevance I(d_i→x_i|q_i) on learning gain I(x_i→d_i|q_i), because the denominator is the variance of t. But the text and Theorem 2.1 require the reverse: the slope of learning gain on contextual relevance, with denominator var(s). This is not a notational quibble: every LCS value and fitted line in Tables 1, 3, 4, and 9 would change if the regression were transposed. Please either swap the definitions of s_i and t_i in Eq. (3) or explain why the transposed regression is intended; if the latter, the claim that the slope equals p(D|Q)/p(X|Q) in Theorem 2.1 is no longer the quantity being measured.
  2. [Theorem 2.2, Appendix B.4] The proof of Theorem 2.2 is invalid. The marginalization p̂(D̂|Q)=Σ_X p̂(D̂|Q;X) drops the required factor p̂(X|Q); the correct identity is Σ_X p̂(D̂|Q;X)p̂(X|Q). Moreover, the premise p̂(X|Q;D̂) ≤ p̂(X|Q;D*) for all X forces equality pointwise, since both sides sum to 1 over X. Even under a non-literal reading, the condition bounds p̂(X|Q;D), not p̂(D|Q;X), so it cannot imply p̂(D̂|Q) ≤ p̂(D*|Q). A concrete counterexample (e.g., p̂(D̂|Q)=0.8, p̂(D*|Q)=0.2, p̂(X|Q;D̂)=p̂(X|Q;D*)=(0.5,0.5)) satisfies the premise but violates the conclusion. Consequently, Section 3.4's thesis that synthetic LCS lower-bounds real LCS, and the rule 'synthetic LCS > 0.2 implies effective ICL,' are unsupported.
  3. [§2.1–§2.2, Eq. (2)] The paper moves from the loss decrease in Eq. (2), which is log p(D|Q;X) − log p(D|Q), to the 'learning gain' Ip(X→D|Q)=p(D|Q;X)−p(D|Q) in §2.2. These are different quantities, and Theorem 2.1 is proved for the probability difference, not for the log-probability loss decrease. The claim that LCS reflects loss decrease and 'captures continuous loss changes' therefore lacks a derivation; either prove a version of Theorem 2.1 for log-probabilities or consistently redefine the metric as a probability-difference slope and weaken the loss-decrease interpretation.
  4. [§3.2.1, Fig. 2] The threshold LCS=0.2 is selected post hoc from the same scatterplot used to report Pearson r=0.737. With no holdout, cross-validation, or pre-registered rule, the threshold's operational value is overclaimed. The paper should either validate the threshold on independent model-task combinations (e.g., the models in Appendix H.2) or present it as a description of the current data rather than an actionable cutoff.
  5. [§3.3.1, Fig. 4; Theorem 2.1] The attribution analysis is circular: since Theorem 2.1 gives r_p = p(D|Q)/p(X|Q) (up to the least-squares estimate), the two 'key factors' contextual alignment p̂(D|Q) and output calibration p̂(X|Q) are exactly the numerator and denominator of the slope. Figure 4 therefore confirms an algebraic identity rather than providing empirical evidence for these as causes of ICL failure. The paper should present the decomposition as a definitional consequence and seek independent evidence (e.g., interventions on p̂(D|Q) or p̂(X|Q)) for any causal claim.
minor comments (6)
  1. [Throughout] The manuscript contains many typos and grammatical slips, including 'effectivess', 'Accodring', 'insturction', 'abality', 'relevent', 'bewteen', 'to to', and 'statistic'; a thorough proofreading pass is needed.
  2. [§2.2] The acronym is introduced as 'Learning-to-Relevance Ratio (LCS)' in the body, while the title and abstract call it 'Learning-to-Context Slope'; please standardize.
  3. [Appendix B.1] The variable K appears in L_p(X|K;D;Q) in the proof of Eq. (2) but is never defined; this appears to be a typo.
  4. [Table 1 and Figures 6–8] The red highlighting used for low values and negative gains may be ambiguous in grayscale or for color-blind readers; please add symbols or textual markers.
  5. [§3.3.2, Table 3] The section title says shot number does not affect LCS, but Table 3 shows small nonzero changes; the body text says 'little impact', so the title should be softened accordingly.
  6. [Appendix B.3] The assumption r_p ≥ ε(D|Q)/ε(X|Q) ≥ ε(D|Q;X)/ε(X|Q;D) is stated without justification; it should either be proved or explicitly flagged as an assumption that limits the error-robustness conclusion.

Circularity Check

2 steps flagged · score 5.0 of 10

LCS's attribution of ICL failures to two 'key factors' is true by construction: Theorem 2.1 defines the LCS slope as p(D|Q)/p(X|Q), the exact numerator and denominator that Figure 4 then 'discovers.' Appendix B.3 assumes the error ordering it claims to prove, and Theorem 2.2's proof is invalid, leaving the label-free lower-bound claim without support.

  1. self definitional [Section 2.2, paragraph after Eq. (3); Figure 4 in Section 3.3.1]
    "Based on Theorem 2.1, it can be observed that there are two main factors influencing the ICL effectiveness: the contextual alignment capability that learn the question-relevant information from the demonstrations ( ˆp(D|Q)), and the output calibration capability that verify the correctness of the output to the given input (ˆp(X|Q))."

    Theorem 2.1 states Ip(X→D|Q) = (p(D|Q)/p(X|Q)) · Ip(D→X|Q), and LCS is defined as the least-squares slope between these two quantities. Hence p(D|Q) and p(X|Q) are, by construction, the numerator and denominator of the very slope used to define LCS. The paper's 'discovery' that ICL effectiveness is governed by weak contextual alignment or strong output calibration is therefore a restatement of the definition, not an empirically established causal attribution. Figure 4 then measures p_hat(D|Q), p_hat(X|Q), and LCS on the same data, so the supposed attribution analysis is entailed by the metric's construction rather than providing independent evidence.

  2. other [Appendix B.3 (Error of Theorem 2.1)]
    "We suppose that rp ≥ ε(D|Q)/ε(X|Q) ≥ ε(D|Q;X)/ε(X|Q;D), i.e., the error growth rate with introduced demonstrations is smaller than that without demonstrations, which is further smaller than the ICL effectiveness."

    The appendix aims to prove that the slope-based estimate r_hat has smaller error than the ratio p_hat(D|Q)/p_hat(X|Q), concluding ΔI ≤ Δp. But the 'suppose' condition already assumes the desired error-growth ordering: it states that the demonstration-conditioned error ratio is smaller than the unconditioned error ratio, which is exactly the ordering needed to make the slope estimate more accurate. The proof therefore assumes the very conclusion it purports to establish, so the claimed error robustness of LCS is not an independent result.

full rationale

The core definition of LCS is not itself circular: LCS is a least-squares slope fitted to learning-gain versus contextual-relevance data, and the reported Pearson correlation of 0.737 with performance change is an external, empirical check. However, the paper's central attribution claim is circular by construction. Because Theorem 2.1 makes the true slope equal to p(D|Q)/p(X|Q), the two 'key factors' named in Section 2.2 and then 'analyzed' in Figure 4 are just the numerator and denominator of the metric itself; the attribution is a tautology rather than a measured discovery. The Appendix B.3 error analysis is also circular in a narrower sense: it assumes the error-growth ordering that it claims to prove. Separately, but importantly for the label-free claim, Theorem 2.2 in Section 2.3 and Appendix B.4 is invalid as a derivation: the proof marginalizes p_hat(D_hat|Q) without the required p_hat(X|Q) factor, and the premise p_hat(X|Q;D_hat) ≤ p_hat(X|Q;D*) does not imply p_hat(D_hat|Q) ≤ p_hat(D*|Q). This is a correctness failure rather than a circularity step, but it means the synthetic-data lower-bound rule (LCS > 0.2 implies effective ICL) has no valid theoretical support. Overall, the paper has genuine independent empirical content, so the score is moderate rather than extreme, but the headline attribution and the error-bound proof reduce to their own assumptions.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on fitted quantities (the 0.2 threshold), background assumptions inherited from prior work (loss as effectiveness measure), and paper-specific assumptions that lack independent support (error-ordering condition, paraphrasing validity, real-demonstration superiority). No new entities are postulated.

free parameters (1)
  • LCS empirical threshold = 0.2
    Chosen from the scatter in Figure 2 to separate positive from negative performance changes, then used as an actionable threshold in Section 3.4. It is fit to the same data used for validation, making it a free parameter.
assumptions (5)
  • domain assumption ICL effectiveness can be measured by generation loss (negative log-likelihood), per [51, 58]
    Invoked in Section 2.1 to motivate LCS as a loss-decrease measure. This is a background assumption from prior work, stated but not independently justified.
  • ad hoc to paper Learning gain (probability difference) is interchangeable with loss decrease (log-probability difference)
    Stated in Section 2.2 without proof. The identity in Eq. (2) is about log-probability differences, but the paper defines learning gain as a probability difference and builds Theorem 2.1 on it.
  • ad hoc to paper Error growth rate ordering r_p >= epsilon(D|Q)/epsilon(X|Q) >= epsilon(D|Q;X)/epsilon(X|Q;D)
    Assumed in Appendix B.3 to show the empirical slope has smaller error than the ratio of marginals. No empirical justification is given for this ordering.
  • domain assumption Real demonstrations always better assist the model than synthetic demonstrations (Theorem 2.2 condition)
    Interpreted in Section 2.3 as 'real demonstrations always better assist in correctly answering the question compared to synthetic demonstrations.' This is a strong assumption about demonstration quality.
  • ad hoc to paper Paraphrasing removes format-induced biases in likelihood calculation
    Assumed in Appendix C. If false, likelihood differences reflect stylistic variation rather than knowledge gain, and LCS is confounded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusions." pith.science (2026). https://pith.science/paper/R2JXVGNP

@misc{pith2026250623146,
  author       = {Pith},
  title        = {Pith review of: Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R2JXVGNP}},
  note         = {Machine review of arXiv:2506.23146}
}
read the original abstract

In-context learning (ICL) has emerged as an effective approach to enhance the performance of large language models (LLMs). However, its effectiveness varies significantly across models and tasks, posing challenges for practitioners to determine when ICL reliably improves performance. Current evaluation approaches, reliant on performance change after applying ICL, suffer from low reliability, poor attribution, and impracticality in data-insufficient scenarios. We propose the Learning-to-Context Slope (LCS), a novel metric that quantifies ICL effectiveness by modeling the slope between learning gain (loss decrease from demonstrations) and contextual relevance (demonstration-input relevance). LCS addresses key limitations of performance-based metrics: (1) it captures continuous loss changes even when outputs are incorrect, improving reliability; (2) its formulation attributes ICL failures to weak contextual alignment (inability to adapt inputs to demonstrations) or strong output calibration (self-verification of correctness); and (3) it minimizes reliance on labeled data via synthetic evaluation. Extensive experiments demonstrate that LCS strongly correlates with performance improvements in labeled settings and reliably reflects true effectiveness in biased or data-scarce scenarios. Further analysis reveals actionable thresholds for LCS and identifies model capabilities critical to ICL success.

Figures

Figures reproduced from arXiv: 2506.23146 by the authors.

Figure 1
Figure 1. (a) Performance change of different models before and after applying ICL, where ICL [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. The experimental results of using Llama3.1-8b on GSM8K and Amazon under the full set [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 2
Figure 2. The performance improvement ∆ brought by ICL (y-axis) with different LCS (x-axis) on different models and datasets. The solid line in the graph represents the fitted line for all data points. The two dashed lines rep￾resent LCS = 0.2 and ∆ = 0, respectively. The performance improvement brought by ICL is positively related to LCS. To evaluate whether LCS effectively reflects the efficacy of ICL, we analyze the perfor… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The results of the contextual alignment capability (pˆ(D|Q)), the output calibration ca￾pability (pˆ(X|Q)) and LCS of Llama2-7b and Llama3.1-8b on GSM8K and Amazon Review. pˆ(D|Q) and pˆ(X|Q) are calculated as the aver￾age value on all test data. In §2.2, we discuss th…
Figure 5
Figure 5. Figure 5: The intercept of the fitted line on each dataset and each model. We also compare the intercepts of Llama3.1 under different scales in Appendix H.3. Apart from the slope, the intercept of the fitted line also reflects the effectiveness of ICL under different settings. W…
Figure 6
Figure 6. Figure 6: The variation of Ipˆ(X → D|Q) (y-axis) with Ipˆ(D → X|Q) (x-axis) on different datasets using Llama2-7b. The title of each plot displays the corresponding dataset and fitted line. Each blue dot in the plot represents a data point, and the red line indicates the fitted …
Figure 7
Figure 7. Figure 7: The variation of Ipˆ(X → D|Q) (y-axis) with Ipˆ(D → X|Q) (x-axis) on different datasets using Llama3.1-8b. The legend is the same as [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: The variation of Ipˆ(X → D|Q) (y-axis) with Ipˆ(D → X|Q) (x-axis) on different datasets using Llama-R1-8b. The legend is the same as [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: The variation of the learning gain (y-axis) with different similarity metrics (x-axis) on [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: The intercepts of the fitted lines of Llama3.1-8b and Llama3.1-70b. [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 53 canonical work pages

  1. [1]

    Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle

    Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie C.Y . Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. In ICML 2024 Workshop on In-Context Learning, 2024

    work page 2024

  2. [2]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021

    2021

  3. [3]

    Broder, Steven C

    Andrei Z. Broder, Steven C. Glassman, Mark S. Manasse, and Geoffrey Zweig. Syntactic clustering of the web. Computer Networks and ISDN Systems, 29(8):1157–1166, 1997. Papers from the Sixth International World Wide Web Conference

    work page 1997

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott G...

    1901

  5. [5]

    Selective demonstrations for cross-domain text-to- SQL

    Shuaichen Chang and Eric Fosler-Lussier. Selective demonstrations for cross-domain text-to- SQL. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14174–14189, Singapore, December 2023. Association for Computational Linguistics

    work page 2023

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  7. [7]

    Self-ICL: Zero-shot in- context learning with self-generated demonstrations

    Wei-Lin Chen, Cheng-Kuang Wu, Yun-Nung Chen, and Hsin-Hsi Chen. Self-ICL: Zero-shot in- context learning with self-generated demonstrations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15651–15662, Singapore, December 2023. Association for Computationa...

    work page 2023

  8. [8]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023

    work page 2023

Show all 64 references
  1. [9]

    Finqa: A dataset of numerical reasoning over financial data

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. Finqa: A dataset of numerical reasoning over financial data. Proceedings of EMNLP 2021, 2021

    2021

  2. [10]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021

    2021

  3. [11]

    Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V . Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context, 2019. 10

    2019

  4. [12]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025

  5. [13]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conferenc...

    2024

  6. [15]

    Edelman, eran malach, and Surbhi Goel

    Ezra Edelman, Nikolaos Tsilivis, Benjamin L. Edelman, eran malach, and Surbhi Goel. The evolution of statistical induction heads: In-context learning markov chains. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  7. [16]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, A...

    2024

  8. [17]

    Understanding in-context learning via supportive pretraining data

    Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. Understanding in-context learning via supportive pretraining data. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the As...

    2023

  9. [18]

    Structured prompting: Scaling in-context learning to 1,000 examples, 2022

    Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples, 2022

    2022

  10. [19]

    Self-demos: Eliciting out-of-demonstration generalizability in large language models

    Wei He, Shichun Liu, Jun Zhao, Yiwen Ding, Yi Lu, Zhiheng Xi, Tao Gui, Qi Zhang, and Xuanjing Huang. Self-demos: Eliciting out-of-demonstration generalizability in large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for C...

    2024

  11. [20]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track ...

    2021

  12. [21]

    Explainable sentiment analysis with deepseek-r1: Perfor- mance, efficiency, and few-shot learning, 2025

    Donghao Huang and Zhaoxia Wang. Explainable sentiment analysis with deepseek-r1: Perfor- mance, efficiency, and few-shot learning, 2025

    2025

  13. [22]

    Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator

    Gyeonghary Kim, Won Ik Cho, Seok Min Shin, Sang-goo Lee, and Jamin Kim. Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator. arXiv preprint arXiv:2206.08082, 2022

    2022 arXiv

  14. [23]

    In-context learning learns label relationships but is not conventional learning

    Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but is not conventional learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  15. [24]

    Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from explanations in context? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Findings of the Asso...

    2022

  16. [25]

    Diverse demonstrations improve in-context compositional generalization

    Omer Levy, Gabriel Poesia, Sewon Min, Romain Paulus, Luke Zettlemoyer, and Mike Lewis. Diverse demonstrations improve in-context compositional generalization. arXiv preprint arXiv:2211.12703, 2022. 13

    2022 arXiv

  17. [26]

    Patil, Matei Zaharia, Joseph E

    Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G. Patil, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Llms can easily learn to reason from demonstrations structure, not content, is what matters!, 2025

    2025

  18. [27]

    Self-prompting large language models for zero-shot open-domain QA

    Junlong Li, Jinyuan Wang, Zhuosheng Zhang, and Hai Zhao. Self-prompting large language models for zero-shot open-domain QA. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computat...

    2024

  19. [28]

    Unified demonstration retriever for in-context learning

    Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. Unified demonstration retriever for in-context learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Ass...

    2023

  20. [29]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2024

    2024

  21. [30]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  22. [31]

    Dr.icl: Demonstration-retrieved in-context learning, 2023

    Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y Zhao. Dr.icl: Demonstration-retrieved in-context learning, 2023

    2023

  23. [32]

    In-context learning with retrieved demonstrations for language models: A survey

    Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. In-context learning with retrieved demonstrations for language models: A survey. Transactions on Machine Learning Research, 2024. Survey Certification

    2024

  24. [33]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Proceedings of the 2022 Conference ...

    2022

  25. [34]

    Differential learning kinetics govern the transition from memorization to generalization during in-context learning

    Alex Nguyen and Gautam Reddy. Differential learning kinetics govern the transition from memorization to generalization during in-context learning. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  26. [35]

    Justifying recommendations using distantly- labeled reviews and fine-grained aspects

    Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fine-grained aspects. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Pr...

    2019

  27. [36]

    Competition dynamics shape algorithmic phases of in-context learning

    Core Francisco Park, Ekdeep Singh Lubana, and Hidenori Tanaka. Competition dynamics shape algorithmic phases of in-context learning. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  28. [37]

    In-context learning with iterative demonstration selection

    Chengwei Qin, Aston Zhang, Chen Chen, Anirudh Dagar, and Wenming Ye. In-context learning with iterative demonstration selection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7441–745...

    2024

  29. [38]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025

  30. [39]

    Pretraining task diversity and the emergence of non-bayesian in-context learning for regression

    Allan Raventos, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  31. [40]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389, April 2009

    2009

  32. [41]

    Learning to retrieve prompts for in-context learning

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for...

    2022

  33. [42]

    Ordered neurons: Inte- grating tree structures into recurrent neural networks

    Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Inte- grating tree structures into recurrent neural networks. In International Conference on Learning Representations, 2019

    2019

  34. [43]

    Amit Singhal and I. Google. Modern information retrieval: A brief overview. IEEE Data Engineering Bulletin, 24, 01 2001

    2001

  35. [44]

    Demonstration augmen- tation for zero-shot in-context learning

    Yi Su, Yunpeng Tai, Yixin Ji, Juntao Li, Yan Bowen, and Min Zhang. Demonstration augmen- tation for zero-shot in-context learning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 14232– 14244...

    2024

  36. [45]

    Large language models for data annotation and synthesis: A survey

    Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedi...

    2024

  37. [46]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023

  38. [47]

    Active learning principles for in-context learning with large language models

    Tu Vu, Heming Liu, David Dohan, and Denny Zhou. Active learning principles for in-context learning with large language models. In Findings of the Association for Computational Linguis- tics: EMNLP 2023, pages 4891–4901. Association for Computational Linguistics, 2023

    2023

  39. [48]

    From few to many: Self-improving many-shot reasoners through iterative optimization and generation

    Xingchen Wan, Han Zhou, Ruoxi Sun, and Sercan O Arik. From few to many: Self-improving many-shot reasoners through iterative optimization and generation. In The Thirteenth Interna- tional Conference on Learning Representations, 2025

    2025

  40. [49]

    Improving demonstration diversity by human-free fusing for text-to-SQL

    Dingzirui Wang, Longxu Dou, Xuanliang Zhang, Qingfu Zhu, and Wanxiang Che. Improving demonstration diversity by human-free fusing for text-to-SQL. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 202...

    2024

  41. [50]

    In-context transfer learning: Demonstration synthesis by transferring similar tasks, 2025

    Dingzirui Wang, Xuanliang Zhang, Qiguang Chen, Longxu Dou, Xiao Xu, Rongyu Cao, YINGWEI MA, Qingfu Zhu, Wanxiang Che, Binhua Li, Fei Huang, and Yongbin Li. In-context transfer learning: Demonstration synthesis by transferring similar tasks, 2025

    2025

  42. [51]

    Loss landscape geometry reveals stagewise development of transformers

    George Wang, Matthew Farrugia-Roberts, Jesse Hoogland, Liam Carroll, Susan Wei, and Daniel Murfet. Loss landscape geometry reveals stagewise development of transformers. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024

    2024

  43. [52]

    Generalized Inverses: Theory and Computa- tions, volume 53 of Developments in Mathematics

    Guorong Wang, Yimin Wei, and Sanzheng Qiao. Generalized Inverses: Theory and Computa- tions, volume 53 of Developments in Mathematics. Springer, Singapore, 2018

    2018

  44. [53]

    Label words are anchors: An information flow perspective for understanding in-context learning

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  45. [54]

    Learning to retrieve in-context examples for large language models

    Liang Wang, Nan Yang, and Furu Wei. Learning to retrieve in-context examples for large language models. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Pap...

    2024

  46. [55]

    Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning

    Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  47. [56]

    MMLU-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language unders...

    2024

  48. [57]

    Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering

    Vikas Yadav, Steven Bethard, and Mihai Surdeanu. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Meth...

    2019

  49. [58]

    In-context learning with representations: Contextual generalization of trained transformers

    Tong Yang, Yu Huang, Yingbin Liang, and Yuejie Chi. In-context learning with representations: Contextual generalization of trained transformers. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024

    2024

  50. [59]

    Representative demonstration selection for in-context learning with two-stage determinantal point process

    Zhao Yang, Yuanzhe Zhang, Dianbo Sui, Cao Liu, Jun Zhao, and Kang Liu. Representative demonstration selection for in-context learning with two-stage determinantal point process. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empiri...

    2023

  51. [60]

    STar: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STar: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022

    2022

  52. [61]

    Is in-context learning sufficient for instruction following in LLMs? In The Thirteenth International Confer- ence on Learning Representations, 2025

    Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Is in-context learning sufficient for instruction following in LLMs? In The Thirteenth International Confer- ence on Learning Representations, 2025

    2025

  53. [62]

    SELF-EXPLAIN: Teaching large language models to reason complex questions by themselves

    Jiachen ZHAO, Zonghai Yao, zhichao Yang, and hong yu. SELF-EXPLAIN: Teaching large language models to reason complex questions by themselves. In R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023

    2023

  54. [63]

    Wong, and Simon See

    Tianshi Zheng, Yixiang Chen, Chengxi Li, Chunyang Li, Qing Zong, Haochen Shi, Baixuan Xu, Yangqiu Song, Ginny Y . Wong, and Simon See. The curse of cot: On the limitations of chain-of-thought in in-context learning, 2025. 16

    2025

  55. [64]

    thinking

    Yuxiang Zhou, Jiazheng Li, Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. The mystery of in-context learning: A comprehensive survey on interpretation and analysis. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empiric...

    2024

  56. [2023]

    Association for Computational Linguistics

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.