Pith. sign in

REVIEW 2 major objections 8 minor 56 references

Uncertainty Unveiled: Can Exposure to More In-context Examples Mitigate Uncertainty for Large Language Models?

T0 review · 2 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that additional in-context examples reduce an LLM's predictive uncertainty chiefly by lowering epistemic uncertainty — the model's lack of task knowledge — and that on hard tasks the reduction appears only after hundreds…

desk verdict Solid descriptive scaling result for many-shot ICL, but the epistemic-uncertainty story is not nailed down by the current estimator. read the letter →

arxiv 2505.21003 v1 pith:4AIZU6J5 submitted 2025-05-27 cs.CL

Yifei Wang , Yu Sheng , Linjing Li , Daniel Zeng This is my paper
classification cs.CL
keywords in-contextlearninguncertaintyquantificationepistemicaleatoricmany-shotlong-contextlanguagemodelslogitanalysistrustworthiness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In-context learning lets a language model solve new tasks by reading worked examples in its prompt, and recent long-context models can now absorb hundreds of them. The paper asks whether that flood of examples does anything for the model's confidence, not just its accuracy, and it answers yes. Across three open seven-billion-parameter models and tasks ranging from sentiment classification to multi-object logical deduction, total predictive uncertainty falls as demonstrations accumulate, and the drop is dominated by epistemic uncertainty — the component that reflects the model's lack of task knowledge — rather than aleatoric uncertainty, the noise inherent in the data. The practical twist is task difficulty: on easy tasks a few examples already do most of the work, while on hard reasoning tasks the uncertainty and accuracy gains only appear once the demonstration count reaches the hundreds, because longer inputs first add noise. The paper also shows the gains come from information, not length: repeating the same examples many times does not reduce uncertainty, only diverse ones do. The reason this matters is that uncertainty is the main trustworthiness signal for deployment; the result says many-shot prompting makes models both more accurate and more dependably confident, and it locates roughly how many examples are needed before that confidence arrives.

What carries the argument

The engine of the analysis is a Bayesian-style uncertainty decomposition adapted from a few-shot ICL framework. For each shot count, the model runs on L=6 randomly sampled demonstration sets, and for each set the answer distribution is estimated from m=10 beam-search generations; the average of those distributions gives total uncertainty $TU = H(\sigma(\sum_j A_{j,:}))$, the average of the per-set entropies gives epistemic uncertainty $EU = \frac{1}{L}\sum_j H(\sigma(A_{j,:}))$, and aleatoric uncertainty is the difference $AU = TU - EU$. This split carries the whole argument: EU is interpreted as the model's missing task knowledge, so a falling EU with more examples is direct evidence of knowledge injection. The supporting mechanism comes from residual-stream projection: each layer's residual state at the answer position is passed through the unembedding matrix into vocabulary space, letting the authors watch confidence build layer by layer and see many-shot ICL concentrate logit mass on the correct symbol.

What would settle it

Take a hard task at a shot count where the paper reports a large EU drop (e.g., 240-shot logical deduction) and rerun the decomposition with demonstration sets drawn from an unrelated task, such as sentiment examples placed in a deduction prompt: if EU falls as steeply with out-of-domain demonstrations as with matched ones, the effect is prompt-structure sensitivity, not task-knowledge injection. A second check: recompute EU with far more than six demonstration sets, and if the values shift materially with L, the quantitative claims rest on an unverified sampling approximation.

Watch

Extended reading notes

Core claim

The paper's central claim is that additional in-context examples reduce total uncertainty in both simple and complex tasks by injecting task-specific knowledge, thereby diminishing epistemic uncertainty and enhancing performance. Using an uncertainty decomposition, the authors show that EU starts as the dominant component of total uncertainty and falls steadily as shots increase, while AU stays low on easy tasks and can even rise on the hardest logical-deduction sets, partially offsetting the EU drop. The mechanism is task knowledge, not context length: repeating a small set of demonstrations many times leaves EU high, whereas the same number of diverse demonstrations lowers it. One model, Qwen1.5-7B, departs from this picture on hard tasks, where 10-shot settings sometimes match 240-shot confidence and accuracy — a variability the paper calls the 'ICL sink'. Finally, projecting residual states from every layer into the vocabulary shows that many-shot ICL concentrates increasing logit mass on the correct option and widens the gap between the correct answer and its distractors, which is the internal signature of the confidence gain.

Load-bearing premise

The argument stands on treating the average output entropy across six randomly sampled demonstration sets as a measure of what the model does not yet know about the task; if that average mostly reflects which examples happened to be sampled, the claim that more examples shrink uncertainty by injecting task knowledge does not follow.

Editorial extensions

If this is right

  • Many-shot ICL improves accuracy and confidence together, so choosing a larger k is a reliability choice, not just a performance choice.
  • On hard reasoning tasks, the uncertainty and accuracy gains appear only after roughly a hundred or more demonstrations; below that, the added context chiefly adds noise.
  • Only diverse demonstrations reduce epistemic uncertainty; repeating the same examples does not, so the benefit is tied to informational content rather than context length.
  • The entropy-based uncertainty measures keep high AUROC as shot counts grow, so they remain trustworthy correctness indicators in long-context settings.
  • Larger models reach lower uncertainty sooner and keep lowering EU with more examples, so the many-shot advantage persists at larger scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If epistemic uncertainty falls through task-knowledge injection, demonstration selection becomes a design lever: choosing examples that maximize coverage of task structure should cut EU faster than random sampling, which is directly testable.
  • The hard-task threshold implies a deployment rule: for each task, locate the shot count at which EU starts to fall before paying for long prompts, because below that threshold extra context buys no confidence.
  • The decomposition averages only six demonstration sets; rerunning it with many more sets and with out-of-domain demonstrations would separate true task-knowledge uncertainty from sensitivity to the particular examples sampled.
  • The result links many-shot ICL to calibration research: if confidence rises in step with accuracy, long-context prompting could act as an implicit calibration mechanism, reducing the need for separate post-hoc recalibration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. This paper studies how the number of in-context demonstrations affects the predictive uncertainty of long-context language models on classification and multiple-choice QA tasks. The authors use three base 7-8B models, plus two larger instruction-tuned models in an appendix, and measure total uncertainty as the entropy of a normalized distribution aggregated over six sampled demonstration sets and ten beam-search generations. Following Ling et al. (2024), they decompose total uncertainty into epistemic (expected entropy per demonstration set) and aleatoric (total minus epistemic) components. Their main empirical findings are: (1) additional examples reduce total uncertainty on easy tasks and, after a critical threshold, on hard BBH-style logical deduction tasks; (2) the reduction is mainly in the epistemic component, while aleatoric uncertainty can rise on hard tasks; (3) repeating the same demonstrations does not reproduce the effect, suggesting information content rather than context length drives it; and (4) layer-wise residual projections show that more examples concentrate logit mass on the correct answer and enlarge the logit gap to distractors. The paper concludes that more examples inject task-specific knowledge, which lowers epistemic uncertainty and improves performance.

Significance. If the epistemic uncertainty estimator is accepted as a measure of task-level knowledge, this is one of the first systematic characterizations of uncertainty dynamics in many-shot ICL, with a useful across-model, across-difficulty experimental scope. The descriptive trends (total uncertainty and accuracy) are credible and consistently reported across models and datasets. The repeated-vs-distant example control in Fig. 7 is a genuine falsifiable test, and the internal logit analysis provides a concrete, if qualitative, mechanism. The paper's main vulnerability is that the EU estimate averages entropy over only six demonstration sets and has no external validation against a known ground-truth task concept; it can therefore be read as measuring prompt sensitivity rather than epistemic uncertainty. This is a load-bearing issue for the abstract's causal claim, but it is addressable with additional controls and a tighter causal framing.

major comments (2)
  1. [§3.3, Fig. 7] The operationalization of epistemic uncertainty as EU = (1/L) Σ_{l=1}^L H(σ(A_{l,:})) — the average over L=6 randomly sampled demonstration sets — measures how concentrated the output distribution is under each particular prompt, not whether the model has acquired the intended task concept. A model that is confidently wrong on every demonstration set would yield low EU, and EU can fall simply because longer contexts make the model more deterministic or less sensitive to the specific demonstrations, without any task-specific knowledge being injected. The repeated-vs-distant control in Fig. 7 is a step in the right direction, but it does not exclude confounds such as label-distribution skew, format alignment, or spurious correlations in the example content. Because the abstract's causal narrative ('injecting task-specific knowledge, thereby diminishing EU') depends on EU being a faithful proxy for task-relevant knowledge, please either add a random-label (label-shuffled) control and show that EU does not fall under it, or soften the causal claim to a descriptive one.
  2. [§4.1, Figs. 4 and 8] All decomposition results are computed with L=6 demonstration sets and m=10 beam-search sequences, and the reported error bands are over three runs only. Since the central quantity EU is an average over demonstration sets, the relevant sampling variance is over the choice of demonstrations; with L=6, the apparent decline in EU between, say, 20-shot and 40-shot in Fig. 8 could be within sampling noise. Please report bootstrap confidence intervals over randomly resampled demonstration sets (or a larger L) and test the monotonicity of the EU decrease; without this, the claim that the TU reduction is 'primarily attributed to a decline in EU' is not statistically supported.
minor comments (8)
  1. [§3.2] The first display of TU is written as an unnormalized sum over demonstration sets, and the role of σ is unclear; please define A_{l,y} and the normalization explicitly before using the abbreviation in Sec. 3.3.
  2. [Fig. 5] The right panel of Fig. 5 has a y-axis label reading 'TU' although the figure reports accuracy; please correct the label to 'Accuracy'.
  3. [Tables 2, 11, 12] The threshold τ in |∆U| > τ is never defined; please state its value and how it was chosen, and clarify the two-row layout in the table captions.
  4. [§4.2] The term 'ICL sink' is introduced without a formal definition; please define it precisely and indicate whether it is meant descriptively or as an explanatory mechanism.
  5. [Table 3] The caption 'Average logit difference / the largest logit' is ambiguous; specify whether the difference is between the correct and the second-highest logit and whether the values are averaged over all items.
  6. [§5.1] The residual-stream projection is described with unnumbered equations, and the sentence 'there existed no principled method existed' (Sec. 3.2) contains a grammatical error; also 'invertigation' should be 'investigation'.
  7. [Appendix G] The statement 'We used chatgpt to assist with correcting spelling errors in writing' should specify the tool and the scope of use, following common disclosure guidelines.
  8. [References] Several references are incomplete or inconsistently formatted, e.g., 'Akyürek et al.' lacks a year and 'He et al.; Li and Papyan' appears without years in §5.1; please standardize.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the uncertainty decomposition is externally sourced and directly measured, the central claim is empirically anchored, and the sole self-citation is not load-bearing.

full rationale

The paper's central claim is an empirical measurement rather than a derivation: total, epistemic, and aleatoric uncertainty are computed from model logits under increasing shot counts, and no parameter is fitted to the outcome it later describes as a finding. The EU/AU decomposition is imported from Ling et al. (2024), an external framework with no author overlap, so the decomposition is independent evidence rather than a self-citation. The only self-citation (Wang et al., 2024) appears in the Introduction as a general pointer to trustworthiness research; it does not support the uncertainty decomposition or the shot-count results. The repeated-versus-distant demonstration control in Figure 7 provides a genuine experimental contrast: repeating identical demonstrations does not reduce EU, while diverse examples do, so the conclusion that information content rather than context length drives the reduction is falsifiable and not definitionally forced. The skeptical concern that the EU estimator may reflect prompt sensitivity rather than task knowledge is a construct-validity limitation, not a circularity: if the estimator were invalid, the empirical curves would be uninformative, but they would not be true by definition. Accordingly, no circular step is present; the score of 2 reflects only the presence of a minor, non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central conclusions rest on domain assumptions imported from prior work: that ICL can be described by a latent task concept beta and that entropy computed from sampled demonstrations is a faithful decomposition. The paper does not fit any free parameters to make results appear, and no genuinely new entities are introduced; the term ICL sink is a naming of an observed pattern rather than a new mechanism.

assumptions (4)
  • domain assumption The Bayesian ICL model with a latent task concept beta (Ling et al. 2024) correctly describes how demonstrations affect predictive distributions in many-shot ICL.
    Invoked in Sec 3.3 to define EU and AU; no independent verification that beta captures task-specific knowledge at hundreds of shots.
  • domain assumption Sampling six demonstration sets and ten beam-search outputs yields an accurate approximation of the predictive uncertainty components.
    Sec 4.1 states these numbers; the approximation error is not quantified.
  • domain assumption Logit-based probabilities over candidate answers are adequate proxies for confidence despite known model miscalibration.
    Sec 3.2 acknowledges calibration issues and aggregates probabilities; validation is only indirect via AUROC.
  • domain assumption Projecting intermediate residual streams through the unembedding matrix reveals internal confidence at each layer.
    Sec 5.1 follows the tuned lens literature, but the authors do not validate that projected probabilities faithfully reflect confidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty Unveiled: Can Exposure to More In-context Examples Mitigate Uncertainty for Large Language Models?." pith.science (2026). https://pith.science/paper/4AIZU6J5

@misc{pith2026250521003,
  author       = {Pith},
  title        = {Pith review of: Uncertainty Unveiled: Can Exposure to More In-context Examples Mitigate Uncertainty for Large Language Models?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4AIZU6J5}},
  note         = {Machine review of arXiv:2505.21003}
}
read the original abstract

Recent advances in handling long sequences have facilitated the exploration of long-context in-context learning (ICL). While much of the existing research emphasizes performance improvements driven by additional in-context examples, the influence on the trustworthiness of generated responses remains underexplored. This paper addresses this gap by investigating how increased examples influence predictive uncertainty, an essential aspect in trustworthiness. We begin by systematically quantifying the uncertainty of ICL with varying shot counts, analyzing the impact of example quantity. Through uncertainty decomposition, we introduce a novel perspective on performance enhancement, with a focus on epistemic uncertainty (EU). Our results reveal that additional examples reduce total uncertainty in both simple and complex tasks by injecting task-specific knowledge, thereby diminishing EU and enhancing performance. For complex tasks, these advantages emerge only after addressing the increased noise and uncertainty associated with longer inputs. Finally, we explore the evolution of internal confidence across layers, unveiling the mechanisms driving the reduction in uncertainty.

Figures

Figures reproduced from arXiv: 2505.21003 by the authors.

Figure 1
Figure 1. Humans tend to gain task-specific knowledge [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The sources of AU and EU in many-shot ICL. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A workflow for uncertainty quantification and decomposition under many-shot ICL settings, involves [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: The average TU under k-shot ICL with error bands for three runs. 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 Llama-3.1-8B 0.00 0.25 0.50 0.75 1.00 Acc 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 Mistral-7B-v0.2 0.00 0.25 0.50 0.75 1.00 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 Qwen1.5-7B 0.00 0.25 0.50…
Figure 5
Figure 5. Figure 5: The average accuracy under k-shot ICL with error bands for three runs. indicate a simultaneous rise in accuracy and con￾fidence as more in-context examples are provided, highlighting the correlation between improved con￾fidence and performance gains for LLMs. For *easy…
Figure 6
Figure 6. Figure 6: TU distribution of 2000 examples under cer [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: EU of Llama-3.1-8B on AG News and log￾ical_deduction_five_objects datasets for distant exam￾ples vs. repeating 4/10 examples N times. pacity. This underscores the architectural strengths of Mistral, which leverages a sparse Mixture of Experts (MoEs) (Shazeer et al., 20…
Figure 8
Figure 8. Figure 8: Uncertainty decomposition results for both easy mode (left) and hard mode (right). [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Average probabilities of Mistral-7B-v0.2 on the Commonsense QA dataset for MCQA items where the [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Average logits of Mistral-7B-v0.2 on the Commonsense QA dataset for MCQA items with the correct [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 12
Figure 12. Figure 12: The inner confidence changes (0-1 probability) of five options ["A", "B", "C", "D", "E"] for a specific [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 14
Figure 14. Figure 14: Average logits and probabilities of Mistral-7B-v0.2 on the Commonsense QA dataset for MCQA items [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Prompt template with a test input for AG News dataset. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Prompt template with a test input for SST-2 dataset. [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Prompt template with a test input for Commonsense QA dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Prompt template with a test input for logical deduction three objects dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]
Figure 19
Figure 19. Figure 19: Prompt template with a test input for logical deduction five objects dataset. [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: Prompt template with a test input for logical deduction seven objects dataset. [PITH_FULL_IMAGE:figures/full_fig_p020_20.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 14 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle

    Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie C.Y. Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle. 2024. https://openreview.net/forum?id=goi7DFHlqS Many-shot in-context learning . In ICML 2024 Workshop on In-Context Learning

    work page 2024

  4. [4]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Aky \"u rek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations

    work page

  5. [5]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  6. [6]

    Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. 2024. https://doi.org/10.18653/v1/2024.acl-long.419 MARS : Meaning-aware response scoring for uncertainty estimation in generative LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1...

    doi:10.18653/v1/2024.acl-long.419 2024

  7. [7]

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112

    arXiv 2023

  8. [8]

    Gormley, and Graham Neubig

    Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. 2024. https://openreview.net/forum?id=4KAmc7vUbq In-context learning with long-context models: An in-depth exploration . In First Workshop on Long-Context Foundation Models @ ICML 2024

    2024

Show all 56 references
  1. [9]

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. 2024. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI...

    2024

  2. [10]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey W...

    2020

  3. [11]

    Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022. Data distributional properties drive emergent in-context learning in transformers. Advances in neural information processing systems, 35:18878--18891

    2022

  4. [12]

    Marta R Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv e-prints, pages arXiv--2207

    2022

  5. [13]

    Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. https://doi.org/10.18653/v1/2023.acl-long.893 Analyzing transformers in embedding space . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16124-...

    2023 doi

  6. [14]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.64 A survey on in-context learning . In Proceedings of the 2024 Conference on E...

    2024 doi

  7. [15]

    Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: core tasks, applications and evaluation. J. Artif. Int. Res., 61(1):65–170

    2018

  8. [16]

    Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, Muhammad Shahzad, Wen Yang, Richard Bamler, and Xiao Xiang Zhu. 2022. http://arxiv.org/abs/2107.03342 A surve...

    2022 arXiv

  9. [17]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.446 Transformer feed-forward layers are key-value memories . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484--5495, ...

    2021 doi

  10. [18]

    Albert Gu and Tri Dao. 2024. https://openreview.net/forum?id=AL1fq05o7H Mamba: Linear-time sequence modeling with selective state spaces

    2024

  11. [19]

    Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M

    Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. https://doi.org/10.18653/v1/2021.findings-acl.413 XL -sum: Large-scale multilingual abstractive summarization for 44 languages . In Fin...

    2021 doi

  12. [20]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition

  13. [21]

    Wenchong He, Zhe Jiang, Tingsong Xiao, Zelin Xu, and Yukun Li. 2023. A survey on uncertainty quantification methods for deep learning. arXiv preprint arXiv:2302.13425

    2023

  14. [22]

    Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2024. https://openreview.net/forum?id=byxXa99PtF Decomposing uncertainty for large language models through input clarification ensembling . In Forty-first International Conference on Machine Learning

    2024

  15. [23]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  16. [24]

    Yixing Jiang, Jeremy Andrew Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y. Ng. 2024. https://openreview.net/forum?id=j2rKwWXdcz Many-shot in-context learning in multimodal foundation models . In ICML 2024 Workshop on In-Context Learning

    2024

  17. [25]

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. http://arxiv.org/abs/2401.01325 Llm maybe longlm: Self-extend llm context window without tuning

    2024 arXiv

  18. [26]

    Aditya Prakash, and Chao Zhang

    Lingkai Kong, Harshavardhan Kamarthi, Peng Chen, B. Aditya Prakash, and Chao Zhang. 2023. https://doi.org/10.1145/3580305.3599577 Uncertainty quantification in deep learning . In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, pag...

    2023

  19. [27]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. https://openreview.net/forum?id=VD-AYtP0dve Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation . In The Eleventh International Conference on Learning Representations

    2023

  20. [28]

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30

    2017

  21. [29]

    Jianing Li and Vardan Papyan. 2023. https://openreview.net/forum?id=zOCIKYVaF5 Residual alignment: Uncovering the mechanisms of residual networks . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  22. [30]

    Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024. http://arxiv.org/abs/2404.02060 Long-context llms struggle with long in-context learning

    2024 arXiv

  23. [31]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. https://openreview.net/forum?id=DWkJCSxKU5 Generating with confidence: Uncertainty quantification for black-box large language models . Transactions on Machine Learning Research

    2024

  24. [32]

    Chen Ling, Xujiang Zhao, Xuchao Zhang, Wei Cheng, Yanchi Liu, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Jie Ji, Guangji Bai, Liang Zhao, and Haifeng Chen. 2024. https://doi.org/10.18653/v1/2024.naacl-long.184 Uncertainty quantification for in-context learning of la...

    2024 doi

  25. [33]

    Xin Liu, Muhammad Khalifa, and Lu Wang. 2024. https://openreview.net/forum?id=jH67LHVOIO Litcab: Lightweight language model calibration over short- and long-form responses . In The Twelfth International Conference on Learning Representations

    2024

  26. [34]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.759 Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on ...

    2022 doi

  27. [35]

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bart omiej Koptyra, Hay...

    2023

  28. [36]

    Joshua Robinson and David Wingate. 2023. https://openreview.net/forum?id=yKbprarjc5B Leveraging large language models for multiple choice question answering . In The Eleventh International Conference on Learning Representations

    2023

  29. [37]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024 arXiv

  30. [38]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538

    2017 arXiv

  31. [39]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170 Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on Emp...

    2013

  32. [40]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. https://doi.org/10.1016/j.neucom.2023.127063 Roformer: Enhanced transformer with rotary position embedding . Neurocomput., 568(C)

    2024

  33. [41]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. https://doi.org/10.18653/v1/2023.findings-acl.824 Challenging BIG -bench tasks and whether chain-of-thought c...

    2023 doi

  34. [42]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  35. [43]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  36. [44]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. http://arxiv.org/abs/2302.13971 Llama: ...

    2023 arXiv

  37. [45]

    Matias Valdenegro-Toro and Daniel Saromo Mori. 2022. A deeper look into aleatoric and epistemic uncertainty disentanglement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1509--1517

    2022

  38. [46]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  39. [47]

    Yifei Wang, Yuheng Chen, Wanting Wen, Yu Sheng, Linjing Li, and Daniel Dajun Zeng. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.420 Unveiling factual recall behaviors of large language models through knowledge neurons . In Proceedings of the 2024 Conference on Empirical M...

    2024 doi

  40. [48]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022 a . https://openreview.net/forum?id=gEZrGCozdqR Finetuned language models are zero-shot learners . In International Conference on Learning Representations

    2022

  41. [49]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 b . Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  42. [50]

    Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, and Ashish Sabharwal. 2024. http://arxiv.org/abs/2407.15018 Answer, assemble, ace: Understanding how transformers answer multiple choice questions

    2024 arXiv

  43. [51]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. https://openreview.net/forum?id=NG7sS51zVF Efficient streaming language models with attention sinks . In The Twelfth International Conference on Learning Representations

    2024

  44. [52]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809--11822

    2023

  45. [53]

    Guoxin Yu, Lemao Liu, Mo Yu, Yue Yu, and Xiang Ao. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.779 Rethinking the evaluation of in-context learning for LLM s . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14068--14082, ...

    2024 doi

  46. [54]

    Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. 2024 a . https://aclanthology.org/2024.emnlp-main.299 LUQ : Long-text uncertainty quantification for LLM s . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5244--526...

    2024

  47. [55]

    Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Pan, and Lidong Bing. 2024 b . https://doi.org/10.18653/v1/2024.findings-naacl.246 Sentiment analysis in the era of large language models: A reality check . In Findings of the Association for Computational Linguistics: NAACL 2024, pages...

    2024 doi

  48. [56]

    Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. https://proceedings.neurips.cc/paper_files/paper/2015/file/250cf8b51c773f3f8dc8b4be867a9a02-Paper.pdf Character-level convolutional networks for text classification . In Advances in Neural Information Processing Systems, volume 28...

    2015

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.