When Does Few-Shot Prompting Help? A Systematic Empirical Study of Shot-Count Effects Across Model Scale, Architecture, and Output Parsing Robustness

Ashvi Soni; Ayush Dwivedi

REVIEW 6 major objections 4 minor 15 references

When Does Few-Shot Prompting Help? A Systematic Empirical Study of Shot-Count Effects Across Model Scale, Architecture, and Output Parsing Robustness

T0 review · 6 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Few-shot prompting helps some LLMs, hurts others: optimal shot count is model-specific, not predictable from scale.

desk verdict Honest, well-run measurement study whose headline claims are undercut by an internal parser contradiction and a single-prompt-configuration design it frankly acknowledges but never stress-tests. read the letter →

arxiv 2607.22969 v1 pith:EDVOWO27 submitted 2026-07-25 cs.IR

Ayush Dwivedi , Ashvi Soni This is my paper · ORCID

classification cs.IR

keywords few-shotpromptingin-contextlearningshotcounttextclassificationlargelanguagemodelsevaluationmethodologyoutputparsingAGNews

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests five large language models at six different shot counts on a four-class news classification task and finds that adding demonstrations helps, hurts, or does nothing depending on the model. It identifies four behavioral regimes: already-calibrated models with negligible gains, one model that jumps from near-random to strong performance with a single example, a model that degrades monotonically with any examples, and one 70B model whose performance forms a U-shape—best at zero-shot, worst at two shots, partially recovering but never matching zero-shot. The paper also shows that a naive output parser can deflate measured F1 by more than the true effect, up to 206% for one model, making parser choice a first-order methodological concern.

What carries the argument

The central object is the F1-versus-shot-count curve, measured across 30 model–configuration pairs and evaluated with bootstrap confidence intervals, permutation tests, and Cohen's d. The paper interprets the curves through a two-axis taxonomy: strength of the model's zero-shot classification prior versus sensitivity to format disruption from demonstrations. The mechanism proposed for the U-shaped curve is a competition between 'demonstration disruption' (which dominates at low shot counts and triggers deliberative reasoning that degrades label precision) and 'format re-anchoring' (which dominates at higher shot counts and partially restores concise outputs). The corrected regex parser—which

What would settle it

Re-run the same 30 model–shot configurations with demonstrations selected by similarity or with shuffled ordering instead of the fixed class-ordered pool; if the Llama 3.3 70B U-shape (0.907→0.635→0.785) flattens or vanishes under an equally valid demonstration set, the four-regime taxonomy loses its empirical footing.

Watch

Extended reading notes

Core claim

The central claim is that the relationship between shot count and classification performance is not monotonic, not universal, and not predictable from model scale alone. Concretely, across five models and six shot counts (k=0,1,2,3,5,8), the authors observe four distinct regimes: Regime I (Gemini, GPT-4o-mini) shows modest, statistically insignificant gains; Regime II (Llama 3.1 8B) shows catastrophic zero-shot failure (F1=0.525) that recovers dramatically with one shot (F1=0.865, Cohen's d=10.98) before collapsing again at eight shots; Regime III (Llama 4 Scout) is optimal at zero-shot (F1=0.877) and degrades with examples; Regime IV (Llama 3.3 70B) traces a U-curve—0-shot F1=0.907, 2-shot

Load-bearing premise

All five performance curves come from a single fixed demonstration pool with class-ordered presentation and one test seed, so the observed regimes may reflect the specific prompt realization rather than a general law of few-shot behavior.

Editorial extensions

If this is right

Practitioners cannot assume that 3–5 demonstrations is universally best; the optimal shot count ranges from 0 to 8 across the five models tested, and choosing the wrong value can change F1 by as much as 0.34.
For models with strong zero-shot priors, adding any demonstrations can strictly hurt performance, so zero-shot evaluation should always be included as a baseline before paying for few-shot prompts.
Output parsing is not an implementation detail: a broken parser can flip the ranking of models, and parse failure rate should be reported alongside F1 as a standard transparency metric.
Large scale does not predict few-shot benefit; the 70B model gained nothing from demonstrations while the 8B model gained enormously, contradicting the intuition that more capable models benefit more from examples.
A practical decision protocol is suggested: profile zero-shot performance first; if F1 is high, skip few-shot; if low, add one or two examples; always use a robust parser.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the regime taxonomy holds beyond this single dataset and prompt realization, benchmarking claims about 'few-shot performance' of LLMs should be re-stated as conditional on demonstration selection, ordering, and parsing strategy.
The U-shaped curve in Llama 3.3 70B might be specific to four-class topic classification with short labels; testing on tasks with longer label sets or different label semantics would show whether the disruption mechanism generalizes.
An attention-based test of the proposed mechanism is feasible: measuring how much attention the model allocates to demonstrations versus the test article as shot count increases would directly confirm or reject the 'demonstration disruption and format re-anchoring' account.
The cost-latency implication is that zero-shot prompting is not only better but cheaper for Regime III and IV models, so efficient deployment can align with better accuracy.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The measurements look real and the statistical hygiene is good: fixed seed, temperature 0, bootstrap CIs, permutation tests, per-class breakdown, and a limitations section that actually admits the big caveats. The finding that shot-count effects are model-dependent and non-universal is useful, and the Llama 3.3 70B U-curve is a genuinely interesting datapoint, even if the paper's own citations already show optimal-k varies widely. The PFR analysis is a good idea, and the corrected parser is a practical contribution.

But there are two soft spots that matter. First, the parsing-artifact story contradicts its own code. Listing 1 uses `label.lower() in raw.lower()`, and the quoted "corrupted" output contains "World" — it would parse successfully. The paper claims the substring parser fails because the label is preceded by a colon, but substring matching doesn't care about preceding characters. So the mechanism described for the 206% deflation is wrong as written. That doesn't mean parse failures didn't happen; it means the explanation needs to be rewritten, and the example is misleading.

Second, the abstract and conclusion call Llama 4 Scout's behavior "monotonic degradation," but Table 2 shows oscillation: 0.877 -> 0.695 -> 0.820 -> 0.704 -> 0.746 -> 0.695. That's a plain inconsistency.

The bigger structural concern is the one the stress-test flags: all regime curves come from one demonstration pool, one ordering, one seed. The paper acknowledges this in Section 8.1, which is more than many papers do, but the acknowledgment is a paragraph, not a sensitivity analysis. If the 70B U-curve or the Scout oscillation flattens under a different equally valid demonstration set, the four-regime taxonomy and the "not predictable from scale alone" conclusion lose their empirical footing. For a measurement paper, this is manageable as long as the conclusions are framed as single-configuration observations, not general laws.

The mechanistic story about "demonstration-induced reasoning shift" and "format re-anchoring" is speculative, but the paper does label it as a hypothesis and lists attention analysis as future work. That's acceptable.

Who is this for? People who build evaluation pipelines for LLM classification, and anyone doing few-shot benchmarking who wants a cautionary example of how parser choice and demonstration selection can change conclusions. It deserves a serious referee and probably a conditional accept after rewriting the parser section, fixing the monotonic label, and adding a robustness check on demonstration selection or explicitly narrowing the claims.

Referee Report

6 major / 4 minor

Summary. This paper reports a controlled empirical study of five LLMs (Gemini Flash Lite, GPT-4o-mini, Llama 3.1 8B, Llama 3.3 70B, Llama 4 Scout 17B) on AG News (n=200) across six shot counts k ∈ {0,1,2,3,5,8}. It reports macro-F1, 95% bootstrap CIs, permutation-test p-values, and Cohen's d, and claims to identify four behavioral regimes: calibrated performers (Regime I), format-dependent learners (Regime II), zero-shot specialists (Regime III), and a U-shaped curve for Llama 3.3 70B (Regime IV). A second central contribution is the diagnosis and correction of a parsing artifact that allegedly deflated Llama 3.3 70B F1 by up to 206%. The paper concludes that few-shot shot-count effects are not monotonic, not universal across models, and not predictable from model scale alone, and proposes a two-axis taxonomy of few-shot behavior. It also provides practical guidance for model and shot-count selection.

Significance. If the empirical claims hold, the paper would be a useful corrective to the common assumption that more demonstrations are always better or that shot-count effects are scale-driven. The paper has real strengths: it reports point estimates for all 30 configurations, uses bootstrap CIs with B=10,000, gives permutation p-values, and includes an explicit limitations section. The central concept that parser choice can be as consequential as shot count is important for LLM evaluation practice. However, the paper's central qualitative claims rest on a single demonstration configuration and a parser exhibit that is internally inconsistent with its own code, and the statistical machinery for Cohen's d and permutation tests is not clearly valid for macro-F1. These issues must be resolved before the four-regime taxonomy and the U-curve claim can be accepted as established.

major comments (6)

[§5.3 / Listing 1] The claimed parsing failure is contradicted by the provided code. Listing 1 uses `if label.lower() in raw.lower()`, so the representative output 'I would classify this as: World.' would successfully match 'world' and return 'World'. The paper says the parser fails because of a preceding colon/formatting, but the substring parser does not require exact match. As written, the 19.5–76% PFR table and the '206% deflation' calculation are not supported by the exhibit. Either the deployed parser differs from Listing 1 or the actual raw outputs have a different structure; please clarify and provide a reproducible failure example.
[§4.1 / §8.1] The four-regime taxonomy and the central 'not monotonic, not universal, not predictable from scale' conclusion are curve-shape claims, but all curves come from one fixed demonstration configuration: one pool, proportional sampling, class-ordered presentation, and one seed (Section 4.1). Section 8.1 concedes the results 'reflect one specific prompt realization,' and the paper's own citations [7], [8] show demonstration selection/order can shift accuracy by up to ~30%. A sensitivity analysis over at least several independently drawn demonstration sets and orders is needed. Without it, the Llama 3.3 70B U-curve and Llama 4 Scout oscillation may be configuration-specific rather than model-general behaviors.
[§3.4.3 / Eq. (12) / §6.3] The statistical foundation for Cohen's d is not well-defined. Eq. (12) computes d from 'instance-level F1 contributions,' but macro-F1 is an aggregate class-level statistic and is not decomposable into independent per-instance contributions. Similarly, the permutation test in Eq. (11) appears to 'randomly permute predictions across both conditions' in an unpaired manner, which ignores the pairing of test instances across shot conditions. This undermines the validity of d=10.98 and the permutation p-values in Table 5. Please provide a valid paired permutation scheme (e.g., swapping the two predicted labels within each instance) or explicitly justify the unpaired treatment.
[Abstract / §7.1 / §5.2.4] Regime III is described as 'monotonically degrade' in the abstract and conclusion, but the reported Llama 4 Scout data are not monotonic: 0.877 → 0.695 → 0.820 → 0.704 → 0.746 → 0.695. Section 5.2.4 itself calls the curve 'non-monotonic and high-variance.' This is an internal contradiction in the paper's key categorization and should be corrected throughout.
[§3.3.3 / Tables 2 and 3] The corrected F1 estimator defined in Eq. (9) is computed only over the valid instances with successful parses, and Section 5.3 reports residual PFR=2% at k=5,8 for Llama 3.3 70B. Table 2's 'Llama 3.3 70B (fixed)' row therefore mixes F1 over 200 instances for k=0–3 with F1 over ~196 instances for k=5,8. The U-curve's recovery at k=5 (F1=0.785) may not be directly comparable to the zero-shot baseline or to other models. Please report the corrected F1 with residual failures counted as errors as well, or clearly state that the effective sample sizes differ.
[§7.2 / §8.2] The 'demonstration disruption' and 'format re-anchoring' account is presented as a 'mechanistic analysis' (contribution 2, Section 1), but no attention, routing, or generation-level evidence is provided; the paper's own Section 8.2 lists attention analysis as future work. This is a post hoc verbal mechanism. Please either support it with measurements or explicitly label it as a hypothesis and soften the contribution claim.

minor comments (4)

[Eq. (13) / Listing 2] The regex pattern in Eq. (13) is shown as `Science[\/\s&]*Technology`, while Listing 2 has `Science[\/\s&] *Technology` (with a space before `*`). Please make the canonical pattern consistent.
[§5.2.2] GPT-4o-mini is called the 'Monotonic Improver,' but k=1 (0.8248) is below zero-shot (0.8446). This is not strictly monotonic; please qualify the description.
[§6.3 / Table 6] The text reports 'd for 70B zero-shot vs. two-shot (trough) is 8.1' and 'for Scout zero-shot vs. one-shot is 5.3' but these values and their standard errors are not shown. Please include them or omit.
[Global] There are several typos and spacing artifacts, e.g., 'JUL Y 2026' in the header and 'Y et' in the abstract. A careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: inductive measurement study with transparent parser calibration; limitations are robustness concerns, not circularity.

full rationale

This paper is an inductive measurement study rather than a derivation chain, so the central claims (four behavioral regimes, the Llama 3.3 70B U-curve, and the parsing artifact) are direct reports of F1 measurements accompanied by bootstrap CIs, permutation tests, and Cohen's d. No parameter is fitted to a subset of data and then used to predict a closely related quantity: the regex parser in Eq. (13) is an output-extraction tool calibrated by inspecting the model's verbose outputs, and the paper explicitly labels the corrected F1 in Eq. (9) a diagnostic estimator with a selection-bias caveat rather than a primary prediction. The four-regime taxonomy is a post-hoc description of the measured curves, not a fitted re-description disguised as a result, and Section 8.1 honestly concedes that the results reflect one specific prompt realization; that is an external-validity limitation, not circularity. All references are to external prior work; there are no self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The mechanistic 'demonstration disruption / format re-anchoring' account is presented as an interpretation, not as a consequence forced by the equations. Consequently, no load-bearing step reduces to its own inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The paper postulates no new physical entities; its load-bearing inputs are hand-chosen evaluation settings rather than fitted theory. The hand-tuned regex pattern (Eq. 13) and the deterministic demonstration-selection procedure are the two settings most likely to change the reported conclusions if altered. The Section 7.2 mechanism is a post-hoc explanatory construct without independent test in this paper.

free parameters (3)

Robust regex pattern P (Eq. 13) = \b(World|Sports|Business|Science[/\s&]*Technology)\b
Hand-crafted to match the verbose output formats observed from Llama 3.3 70B; it is the instrument that turns PFR 76% to 2% and shapes the corrected F1 curve that defines Regime IV.
Demonstration selection and ordering = two examples per class per shot level; class-ordered; seed 42
Chosen by hand with no ablation (Section 4.1). The paper's own references [7], [8] show such choices can move few-shot accuracy by up to roughly 30%.
Test sample draw = n=200, seed=42 from AG News test split
Single realization; all CIs and regime claims are conditional on this draw. Not a fitted parameter, but the regime claims inherit its specificities.

assumptions (4)

standard math Exchangeability under H0 for permutation tests and percentile bootstrap validity (Eqs. 10-11)
Assumed i.i.d. prediction-level sampling; with a fixed n=200 draw and deterministic API decoding the dependence structure is only approximately standard.
domain assumption Temperature=0 makes repeated identical calls return identical outputs (Section 4.3)
Asserted for all five models including hosted endpoints with unpinned versions ('Gemini Flash Lite Latest'); determinism of proprietary APIs at temperature 0 is not verified.
domain assumption Corrected-F1 estimator is close to true performance for Llama 3.3 70B (PFR <= 2%)
Section 3.3.3 carries a selection-bias caveat. With PFR <=2% the correction is small, but the U-curve claim depends on the corrected parser being valid; at the original parser's 76% PFR the corrected estimator would be ill-defined.
ad hoc to paper Demonstration disruption / format re-anchoring mechanism (Section 7.2)
Postulates two competing forces with no direct evidence in the paper; attention analysis is listed as future work, so the mechanism is currently untested.

invented entities (2)

Four-regime taxonomy (Regimes I-IV)
purpose: Classifies models by their measured shot-count response shape
Post-hoc labels on the 30 measured F1 points; no out-of-sample or cross-dataset test supports the taxonomy as a general law, and it is defined on and only evidenced by the data it summarizes.
Demonstration-induced reasoning shift and format re-anchoring
purpose: Explains the Llama 3.3 70B U-curve mechanism
Proposed in Section 7.2 with no attention or routing measurements; the paper itself defers the test to future work, so the construct is unfalsifiable within this paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of When Does Few-Shot Prompting Help? A Systematic Empirical Study of Shot-Count Effects Across Model Scale, Architecture, and Output Parsing Robustness." pith.science (2026). https://pith.science/paper/EDVOWO27

@misc{pith2026260722969,
  author       = {Pith},
  title        = {Pith review of: When Does Few-Shot Prompting Help? A Systematic Empirical Study of Shot-Count Effects Across Model Scale, Architecture, and Output Parsing Robustness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EDVOWO27}},
  note         = {Machine review of arXiv:2607.22969}
}

read the original abstract

Few-shot prompting, the practice of prepending a small number of input-output demonstration pairs to a query before presenting it to a large language model (LLM), is among the most widely adopted inference-time techniques in NLP. Yet little systematic work investigates how shot count interacts with model scale, architecture, and output format compliance in determining classification performance. This paper presents a controlled study of five LLMs across six shot-count configurations (k in {0,1,2,3,5,8}) on the AG News four-class benchmark (n=200). Our models span proprietary and open-source families: Gemini Flash Lite, GPT-4o-mini, Llama 3.1 8B, Llama 3.3 70B, and Llama 4 Scout 17B. We report macro-averaged F1 with 95% bootstrap confidence intervals (B=10,000), permutation-test p-values, and Cohen's d effect sizes across all 30 configurations. Our findings reveal four qualitatively distinct behavioral regimes: (1) models already well-calibrated at zero-shot that show modest, statistically insignificant gains (Gemini, GPT-4o-mini); (2) models that undergo catastrophic zero-shot failure but recover dramatically with a single example (Llama 3.1 8B, d=10.98, p<0.0001); (3) models optimal at zero-shot that degrade monotonically with additional examples (Llama 4 Scout); and (4) models exhibiting a U-shaped curve (Llama 3.3 70B: 0-shot F1=0.907, 2-shot F1=0.635, 5-shot F1=0.785 with parser corrected). We additionally identify, diagnose, and correct a systematic parsing artifact that artificially deflated Llama 3.3 70B performance by up to 206%, constituting a methodological contribution to LLM evaluation practice. Our results demonstrate that the relationship between shot count and classification performance is not monotonic, not universal, and not predictable from model scale alone.

Figures

Figures reproduced from arXiv: 2607.22969 by the authors.

**Figure 2.** Optimal shot count and corresponding peak F1 for each of the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Performance change relative to each model’s zero-shot baseline, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Macro-averaged F1 with 95% bootstrap confidence intervals [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 7 linked inside Pith

[7]

Calibrate before use: Improving few-shot performance of language models,

Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh, “Calibrate before use: Improving few-shot performance of language models,” inProc. 38th Int. Conf. Machine Learning (ICML), vol. 139, 2021, pp. 12697–12706

2021
[8]

Fan- tastically ordered prompts and where to find them: Overcoming few-shot prompt sensitivity with ensemble transfer learning,

Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P . Stenetorp, “Fan- tastically ordered prompts and where to find them: Overcoming few-shot prompt sensitivity with ensemble transfer learning,” in Proc. 60th Annu. Meeting Assoc. Computational Linguistics (ACL), 2022, pp. 8086–8098

2022
[1]

Language models are few-shot learners,

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P . Dhariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,”Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020

1901
[2]

Scaling laws for neural language models,

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,”arXiv preprint arXiv:2001.08361, 2020

arXiv 2001
[3]

Training compute-optimal large language models,

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., “Training compute-optimal large language models,”arXiv preprint arXiv:2203.15556, 2022

arXiv 2022
[4]

Rethinking the role of demonstrations: What makes in-context learning work?

S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?” inProc. 2022 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2022, pp. 11048– 11064

2022
[5]

An explanation of in-context learning as implicit Bayesian inference,

S. M. Xie, A. Raghunathan, P . Liang, and T. Ma, “An explanation of in-context learning as implicit Bayesian inference,”arXiv preprint arXiv:2111.02080, 2021

arXiv 2021
[6]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,”Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837, 2022

2022

Show all 15 references

[9]

A survey for in-context learning,

Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, L. Li, and Z. Sui, “A survey for in-context learning,”arXiv preprint arXiv:2301.00234, 2022

2022 arXiv
[10]

What makes good in-context examples for GPT-3?

J. Liu, D. Shen, Y. Zhang, W. B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for GPT-3?”arXiv preprint arXiv:2101.06804, 2021

2021 arXiv
[11]

How to fine-tune BERT for text classification?

C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune BERT for text classification?” inChina Nat. Conf. Chinese Computational Linguistics (CCL), Springer, 2023

2023
[12]

Character-level convolutional networks for text classification,

X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,”Advances in Neural Information Processing Systems, vol. 28, 2015

2015
[13]

Holistic evaluation of language models,

P . Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, et al., “Holistic evaluation of language models,”arXiv preprint arXiv:2211.09110, 2022

2022 arXiv
[14]

Self-consistency improves chain of thought reasoning in language models,

X. Wang, J. Wei, D. Schuurmans, Q. V . Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,”arXiv preprint arXiv:2203.11171, 2022. 12 PREPRINT — SUBMITTED TO ARXIV, JUL Y 2026

2022 arXiv
[15]

Lost in the middle: How language models use long contexts,

N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P . Liang, “Lost in the middle: How language models use long contexts,”T ransactions of the Assoc. Computational Linguistics, vol. 12, pp. 157–173, 2024

2024

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [7]

Calibrate before use: Improving few-shot performance of language models,

Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh, “Calibrate before use: Improving few-shot performance of language models,” inProc. 38th Int. Conf. Machine Learning (ICML), vol. 139, 2021, pp. 12697–12706

2021

[2] [8]

Fan- tastically ordered prompts and where to find them: Overcoming few-shot prompt sensitivity with ensemble transfer learning,

Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P . Stenetorp, “Fan- tastically ordered prompts and where to find them: Overcoming few-shot prompt sensitivity with ensemble transfer learning,” in Proc. 60th Annu. Meeting Assoc. Computational Linguistics (ACL), 2022, pp. 8086–8098

2022

[3] [1]

Language models are few-shot learners,

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P . Dhariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,”Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020

1901

[4] [2]

Scaling laws for neural language models,

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,”arXiv preprint arXiv:2001.08361, 2020

arXiv 2001

[5] [3]

Training compute-optimal large language models,

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., “Training compute-optimal large language models,”arXiv preprint arXiv:2203.15556, 2022

arXiv 2022

[6] [4]

Rethinking the role of demonstrations: What makes in-context learning work?

S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?” inProc. 2022 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2022, pp. 11048– 11064

2022

[7] [5]

An explanation of in-context learning as implicit Bayesian inference,

S. M. Xie, A. Raghunathan, P . Liang, and T. Ma, “An explanation of in-context learning as implicit Bayesian inference,”arXiv preprint arXiv:2111.02080, 2021

arXiv 2021

[8] [6]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,”Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837, 2022

2022