Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Evaluating the Sensitivity of LLMs to Prior Context

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Putting long conversational history before a hard multiple-choice question can reduce LLM accuracy by up to 73 percent, and restating the task just before the question recovers most of the loss.

desk verdict Useful empirical study of context sensitivity, but the headline drops are inflated by strict answer-format parsing and the conclusions overstate the case. read the letter →

arxiv 2506.00069 v1 pith:5AS4L5MP submitted 2025-05-29 cs.CL cs.AI

Robert Hankache , Kingsley Nketia Acheampong , Liang Song , Marek Brynda , Raad Khraishi , Greig A. Cowan This is my paper
classification cs.CLcs.AI
keywords largelanguagemodelsmulti-turnevaluationpriorcontextsensitivityGPQAMMLUtaskplacementpromptengineeringanswerformatcompliance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that an LLM's accuracy on hard multiple-choice questions is not a stable property of the model: it depends heavily on the conversational turns that precede the question, even when those turns are unrelated to the task. Using GPQA Diamond questions as targets, the authors insert 4k to 64k tokens of prior context in three styles—real free chat, same-domain STEM question-answer pairs, and cross-domain non-STEM pairs—and measure accuracy against a no-context baseline. They report relative drops as large as 73 percent (Gemini 1.5 Flash in the cross-domain task-at-top condition) and a 32 percent drop for GPT-4o, and they find that placing the task description once at the top is the fragile configuration. If this is right, single-turn scores such as GPQA are not valid proxies for multi-turn deployment, and the simple remedy of restating the task immediately before the target question deserves to be standard practice.

What carries the argument

The central object is a controlled benchmark construction rather than a mathematical identity: each item pairs a GPQA Diamond target question with a prior-context block drawn from LMSYS-Chat-1M free chat or from MMLU question-answer pairs, at four token lengths (4k, 16k, 32k, and 64k). The load-bearing manipulation is task position, task-at-top versus task-repeated, which operationalizes the paper's claim that instructions are diluted by intervening context. Accuracy is scored by a strict parser that accepts only the expected answer format, and the paper's own error analysis separates reasoning errors, format deviations, and non-answers, attributing the largest share of task-at-top failures to formatting deviations.

What would settle it

Re-score the same task-at-top responses counting any answer that contains the correct letter or correct text as correct regardless of format; if the 73 percent and 32 percent relative drops largely disappear, the performance-degradation claim reduces to a format-compliance effect rather than a reasoning loss.

Watch

Extended reading notes

Core claim

The paper's central claim is that prior context degrades LLM multiple-choice performance through a mechanism the authors call task information dilution: when the answer-format instructions appear only at the top of a long context, the model loses track of them and its outputs increasingly fail the expected format, even though the model often selects the correct choice. In the task-at-top experiments, accuracy falls with added context in both same-domain and cross-domain settings, with the largest relative drop—73 percent—observed for Gemini 1.5 Flash after 64k tokens of cross-domain MMLU context, and a 32 percent drop for GPT-4o under the same condition. Repeating the task description immediately before the target query (task-repeated) limits most drops to roughly 15 percent or less and can raise accuracy by a factor of about 3.5 relative to task-at-top. The paper also claims that larger models are not systematically more robust than smaller ones, and that same-domain context is less harmful than cross-domain context, so model rankings and performance expectations derived from single-turn benchmarks should not be trusted in sustained interactions.

Load-bearing premise

The reported accuracy drops assume a strict answer-format parser, and the paper's own Section 4.2.1 finds formatting deviations to be the largest error source in task-at-top settings; if a lenient parser counted the correct choice despite wrong formatting, the headline 73 percent and 32 percent drops would shrink substantially.

Editorial extensions

If this is right

  • Single-turn benchmark scores such as GPQA should not be used to predict accuracy in multi-turn chatbots or long-running agentic systems, because prior context can cut accuracy by more than half for some models.
  • Applications that put instructions in a system prompt or first message should restate the task immediately before the final query, since this recovers most of the lost accuracy and can triple performance.
  • Cross-domain prior turns hurt more than same-domain turns in task-at-top settings, so keeping a conversation in one knowledge domain is itself a mitigation.
  • Model rankings from no-context evaluations can flip once context is added, so model selection should include context-sensitivity measurements rather than single-turn scores.
  • Evaluation pipelines should separate wrong answers from format violations, since the paper's error analysis shows format deviations are the largest source of measured failure in task-at-top conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves untested whether instruction dilution follows a lost-in-the-middle curve; a follow-up that varies the task-description position continuously rather than only at top and bottom would settle whether the effect is monotonic or U-shaped.
  • Because the paper's own error analysis attributes most task-at-top failures to format deviations, a format-agnostic scoring variant would likely show far smaller drops, isolating the true reasoning loss from mere format compliance.
  • Porting the same benchmark construction to models with million-token context windows would test whether degradation continues past 64k tokens or plateaus, as the free-chat curves here suggest.
  • A practical extension not tested here is periodic re-injection of system instructions during long sessions, which the task-repeated results suggest could match the benefit of more expensive memory mechanisms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces context-sensitivity benchmarks built from GPQA target questions preceded by LMSYS-Chat free-chat conversations, MMLU-based multi-turn QA in same- and cross-domain settings, and unrelated English text. Six commercial LLMs (GPT-4o, GPT-4o-mini, Claude Haiku, Claude Sonnet, Gemini Flash, Gemini Pro) are evaluated at context lengths from 4k to 64k tokens under task-at-top and task-repeated prompt placements. The paper reports large accuracy drops with long prior context—up to 73% relative degradation (Gemini Flash, cross-domain, task-at-top, 64k) and 32% for GPT-4o—and finds that repeating the task description before the target query substantially mitigates the drop. The authors conclude that single-turn benchmarks such as GPQA are inadequate proxies for multi-turn deployment performance.

Significance. The study addresses a practically important question: whether single-turn benchmark accuracy transfers to multi-turn and long-context use. The experimental design is systematic: it varies context source, domain match, length, and task position across six models from three providers, and the full per-condition results are tabulated in the appendix. The task-position mitigation result, if it survives re-analysis, is an actionable prompt-engineering insight. However, the central claim about 'performance' degradation is currently confounded with answer-format compliance. The paper's own qualitative analysis identifies formatting deviations as the dominant error source in the task-at-top condition, and the answer-extraction procedure is not specified. The significance of the headline result therefore depends on re-scoring with a lenient parser and separating format adherence from substantive accuracy.

major comments (3)
  1. [Section 4.2.1 and Table 1] The headline drops of 73% and 32% are accuracy scores under answer extraction, but Section 4.2.1 states that the most significant factor in the task-at-top degradation was formatting issues, where the correct choice was selected yet the output format caused a failed extraction. Because the extraction rules are not specified and no code is released, the reader cannot determine what fraction of the drop represents loss of reasoning or knowledge versus format non-compliance. Please re-score all responses with a lenient parser that accepts any output containing the correct answer token or the full choice text, report both strict and lenient accuracy, and quantify the proportion of errors in each of the three categories listed in Section 4.2.1. This is necessary to support the abstract's claim of 'performance drops.'
  2. [Sections 1 and 5] The conclusion that a single-turn benchmark such as GPQA is 'invalid' as a proxy for multi-turn performance is load-bearing and is not supported by the current evidence. If most of the observed drop disappears under lenient scoring, the supported conclusion would be that long task-at-top context degrades format adherence, not task competence. Please either provide the lenient-scoring analysis or weaken the conclusion to match what the strict-format accuracy actually demonstrates.
  3. [Section 3.2 and Section 4.2.1] The task-repeated condition was introduced after observing low performance in task-at-top, and the task description includes the expected answer format. Repeating the task before the query therefore confounds refocusing the model's attention with restating the output format. Since Section 4.2.1 identifies formatting as the dominant error source, the reported mitigation factor of up to 3.5 may be largely a format-compliance effect. Please disentangle these by adding a condition that repeats only the format instruction without the full task description, and by applying the lenient parser to both task-repeated and task-at-top conditions.
minor comments (6)
  1. [Section 3 (Methodology)] Please provide a precise specification of the answer-extraction parser, including examples of accepted and rejected model outputs; the current manuscript describes only that accuracy is measured, with no implementation details.
  2. [Table 4] The rows labeled 'Gemini Flash multi-turn-qa-16k-diff-taskinlast' and 'GPT-4o-mini multi-turn-qa-4k-diff-taskinlast' are not described in the main text; please explain this condition or remove it from the results table.
  3. [Section 4.1 and Appendix D] The unrelated-text experiments show roughly flat performance across context lengths, and for GPT-4o the 4k-token accuracy is above the no-context baseline; the paper's broad phrasing that 'prior context leads to performance degradation' should be qualified as context-type-dependent.
  4. [Figure 3b] The caption and surrounding text do not define the 'recovered drop' quantity used in the y-axis; values below zero and above one are attributed to fluctuations, but the formula should be stated explicitly.
  5. [Appendix D and Section 1] Minor typos include 'bee seen' in Appendix D, 'MobyDick' in Appendix D, and 'in dynamic in real-world applications' in Section 1.
  6. [Section 4.2.1] The statement that formatting issues are the 'most significant factor' is qualitative; please provide a quantitative error decomposition per model and condition so the reader can assess the magnitude of each error source.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a transparent empirical benchmark study with no fitted parameters, no derivation chain that reduces to its inputs, and no load-bearing self-citations.

full rationale

This paper does not contain a derivation chain in the sense the circularity pass targets. Its central claims (accuracy drops up to 73%, factor-of-3.5 recovery from task repetition) are direct arithmetic on measured accuracies reported in Table 1: Gemini Flash falls from 0.476 (no context) to 0.128 (cross-domain, task-at-top, 64k), a 73% relative drop, and task-repeated recovers to 0.438 (a 3.4x ratio). The ground-truth labels come from the external GPQA Diamond benchmark (Section 3.1.1), prior context from external LMSYS-Chat-1M and MMLU corpora (Section 3.1.2), and no parameters are fitted to the data, so no prediction reduces to its input by construction. The paper is transparent about the main construct concern: Section 4.2.1 states that 'the most significant factor identified for the performance degradation under the task-at-top condition was formatting issues,' meaning part of the measured drop reflects strict answer-format extraction rather than reasoning accuracy. This is an acknowledged measurement-validity limitation, not a circular reduction of the conclusion to the metric definition; the metric is fixed before measurement and applied uniformly. The task-repeated condition was indeed introduced after observing low task-at-top performance (Section 3.2), but its results are new independent measurements that could have failed to recover performance, so this is an exploratory-then-confirmatory design choice rather than circularity. There are no self-citations, no imported uniqueness theorems, no ansatz smuggled in via citation, and no renaming of a known result as a new benchmark. The closest concern (parser strictness determining the magnitude of the headline drop) is a correctness/robustness risk, not circularity, and per the review rules does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on experimental design choices (temperature, context lengths, number of runs) and on the validity of the evaluation metric. None of these are fitted parameters in a derivation; they are choices that shape the measured effect sizes.

free parameters (3)
  • temperature = 0.5
    Chosen for all models to balance creativity and determinism (Section 3.3). Affects response variability and hence accuracy measurements.
  • context_length_bins = 4k, 16k, 32k, 64k
    Chosen by hand in Section 3.1.2 and Section 4. The choice of 64k as the longest bin directly determines the headline 73% drop.
  • num_runs = 3 (seeds)
    Three evaluation runs per condition (Section 4). Small number limits statistical power; standard deviations are reported but no significance tests are performed.
assumptions (3)
  • domain assumption GPQA Diamond questions are a valid measure of graduate-level reasoning and knowledge for all six models.
    Section 3.1.1 relies on GPQA Diamond as the target benchmark. If a model has memorized these questions, the baseline and the measured drops would be mis-calibrated.
  • domain assumption The OpenAI tokenizer's token counts are a fair approximation of context length for all models, including Claude and Gemini.
    Footnote 1 states context lengths are computed with the OpenAI tokenizer, but other model families use different tokenizers, so 64k OpenAI tokens may not match 64k tokens in the evaluated model's own vocabulary.
  • domain assumption Strict answer-format compliance is part of task performance, so answers in the wrong format should be scored as incorrect.
    Section 4.2.1 attributes a large share of the degradation to formatting violations, so the accuracy metric assumes that format compliance is part of answering the task correctly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating the Sensitivity of LLMs to Prior Context." pith.science (2026). https://pith.science/paper/5AS4L5MP

@misc{pith2026250600069,
  author       = {Pith},
  title        = {Pith review of: Evaluating the Sensitivity of LLMs to Prior Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5AS4L5MP}},
  note         = {Machine review of arXiv:2506.00069}
}
read the original abstract

As large language models (LLMs) are increasingly deployed in multi-turn dialogue and other sustained interactive scenarios, it is essential to understand how extended context affects their performance. Popular benchmarks, focusing primarily on single-turn question answering (QA) tasks, fail to capture the effects of multi-turn exchanges. To address this gap, we introduce a novel set of benchmarks that systematically vary the volume and nature of prior context. We evaluate multiple conventional LLMs, including GPT, Claude, and Gemini, across these benchmarks to measure their sensitivity to contextual variations. Our findings reveal that LLM performance on multiple-choice questions can degrade dramatically in multi-turn interactions, with performance drops as large as 73% for certain models. Even highly capable models such as GPT-4o exhibit up to a 32% decrease in accuracy. Notably, the relative performance of larger versus smaller models is not always predictable. Moreover, the strategic placement of the task description within the context can substantially mitigate performance drops, improving the accuracy by as much as a factor of 3.5. These findings underscore the need for robust strategies to design, evaluate, and mitigate context-related sensitivity in LLMs.

Figures

Figures reproduced from arXiv: 2506.00069 by the authors.

Figure 1
Figure 1. Experimental LLM performance over variable [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Experimental LLM performance over variable multi-turn [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Performance improvements when repeating the task description before the query question. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: An illustration of varying context scenario in interactive settings where LLMs respond [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Experimental LLM performance, relative to the base [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Experimental LLM performance, relative to the base [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Experimental LLM performance, relative to the base [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Comparing smaller (a) and larger (b) LLM performances in [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Comparing smaller (a) and larger (b) LLM performances in [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Experimental LLM performance over variable [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PersistBench: When Should Long-Term Memories Be Forgotten by LLMs?

    cs.AI 2026-02 conditional novelty 6.0 of 10

    In PersistBench, 18 LLMs failed a median 53% of cross-domain leakage samples and 97% of memory-induced sycophancy samples, with a beneficial-memory failure rate of 16.5%.

Reference graph

Works this paper leans on

46 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Enabling conversational interaction with mobile ui using large language models

    Bryan Wang, Gang Li, and Yang Li. Enabling conversational interaction with mobile ui using large language models. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–17, 2023

    work page 2023

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Empowering education with llms-the next-gen interface and content generation

    Steven Moore, Richard Tong, Anjali Singh, Zitao Liu, Xiangen Hu, Yu Lu, Joleen Liang, Chen Cao, Hassan Khosravi, Paul Denny, et al. Empowering education with llms-the next-gen interface and content generation. InInternational Conference on Artificial Intelligence in Education, pages 32–37. Springer, 2023

    work page 2023

  4. [4]

    Beyond the chat: Executable and verifiable text-editing with llms

    Philippe Laban, Jesse Vig, Marti Hearst, Caiming Xiong, and Chien-Sheng Wu. Beyond the chat: Executable and verifiable text-editing with llms. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–23, 2024

    work page 2024

  5. [5]

    Large language models in education: Vision and opportunities

    Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Jerry Chun-Wei Lin. Large language models in education: Vision and opportunities. In2023 IEEE international conference on big data (BigData), pages 4776–4785. IEEE, 2023. 10

    work page 2023

  6. [6]

    Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue

    Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jia, and Hongying Zan. Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19368–19376, 2024

    work page 2024

  7. [7]

    A survey on recent advances in llm-based multi-turn dialogue systems.arXiv preprint arXiv:2402.18013, 2024

    Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. A survey on recent advances in llm-based multi-turn dialogue systems.arXiv preprint arXiv:2402.18013, 2024

    arXiv 2024

  8. [8]

    True few-shot learning with language models

    Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 34:11054–11070, 2021

    2021

Show all 46 references
  1. [9]

    Needle in the haystack for memory based large language models.arXiv preprint arXiv:2407.01437, 2024

    Elliot Nelson, Georgios Kollias, Payel Das, Subhajit Chaudhury, and Soham Dan. Needle in the haystack for memory based large language models.arXiv preprint arXiv:2407.01437, 2024

    2024 arXiv

  2. [10]

    Cosafe: Evaluating large language model safety in multi-turn dialogue coreference.arXiv preprint arXiv:2406.17626, 2024

    Erxin Yu, Jing Li, Ming Liao, Siqi Wang, Zuchen Gao, Fei Mi, and Lanqing Hong. Cosafe: Evaluating large language model safety in multi-turn dialogue coreference.arXiv preprint arXiv:2406.17626, 2024

    2024 arXiv

  3. [11]

    Fairmt-bench: Benchmarking fairness for multi-turn dialogue in conversational llms.arXiv preprint arXiv:2410.19317, 2024

    Zhiting Fan, Ruizhe Chen, Tianxiang Hu, and Zuozhu Liu. Fairmt-bench: Benchmarking fairness for multi-turn dialogue in conversational llms.arXiv preprint arXiv:2410.19317, 2024

    2024 arXiv

  4. [12]

    A survey on multi-turn interaction capabilities of large language models.arXiv preprint arXiv:2501.09959, 2025

    Chen Zhang, Xinyi Dai, Yaxiong Wu, Qu Yang, Yasheng Wang, Ruiming Tang, and Yong Liu. A survey on multi-turn interaction capabilities of large language models.arXiv preprint arXiv:2501.09959, 2025

    2025 arXiv

  5. [13]

    Agentic ai systems applied to tasks in financial services: Modeling and model risk management crews, 2025

    Izunna Okpala, Ashkan Golgoon, and Arjun Ravi Kannan. Agentic ai systems applied to tasks in financial services: Modeling and model risk management crews, 2025. URL https: //arxiv.org/abs/2502.05439

    2025 arXiv

  6. [14]

    Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  7. [15]

    Gpt-3: Its nature, scope, limits, and consequences

    Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694, 2020

    2020

  8. [16]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  9. [17]

    Llms get lost in multi-turn conversation.arXiv preprint arXiv:2505.06120, 2025

    Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. Llms get lost in multi-turn conversation.arXiv preprint arXiv:2505.06120, 2025

    2025 arXiv

  10. [18]

    Can large language models understand context? In Yvette Graham and Matthew Purver, editors,Findings of the Association for Computational Linguistics: EACL 2024, pages 2004–2018, St

    Yilun Zhu, Joel Ruben Antony Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, and Bo-Hsiang Tseng. Can large language models understand context? In Yvette Graham and Matthew Purver, editors,Findings of the Association for Computational Lin...

    2024

  11. [19]

    Enhancing contextual understanding in large language models through contrastive decoding

    Zheng Zhao, Emilio Monti, Jens Lehmann, and Haytham Assem. Enhancing contextual understanding in large language models through contrastive decoding. In Kevin Duh, He- lena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American Chapter of th...

    2024 doi

  12. [20]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Màrquez, edi- tors,Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pa...

    2019 doi

  13. [21]

    Bigbench: A unified benchmark for social bias in text-to-image generative models based on multi-modal llm.arXiv preprint arXiv:2407.15240, 2024

    Hanjun Luo, Haoyu Huang, Ziye Deng, Xuecheng Liu, Ruizhe Chen, and Zuozhu Liu. Bigbench: A unified benchmark for social bias in text-to-image generative models based on multi-modal llm.arXiv preprint arXiv:2407.15240, 2024

    2024 arXiv

  14. [22]

    TruthfulQA: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, edi- tors,Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Lo...

    2022 doi

  15. [23]

    Inadequacies of large language model benchmarks in the era of generative artificial intelligence.arXiv preprint arXiv:2402.09880, 2024

    Timothy R McIntosh, Teo Susnjak, Nalin Arachchilage, Tong Liu, Paul Watters, and Malka N Halgamuge. Inadequacies of large language model benchmarks in the era of generative artificial intelligence.arXiv preprint arXiv:2402.09880, 2024

    2024 arXiv

  16. [24]

    Benchmarking benchmark leakage in large language models, 2024

    Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models, 2024. URLhttps://arxiv.org/abs/2404.18824

    2024 arXiv

  17. [25]

    MT-eval: A multi-turn capabilities evaluation benchmark for large language models

    Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. MT-eval: A multi-turn capabilities evaluation benchmark for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceeding...

    2024 doi

  18. [26]

    Multi-if: Benchmarking llms on multi-turn and multilingual instructions following.arXiv preprint arXiv:2410.15553, 2024

    Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, et al. Multi-if: Benchmarking llms on multi-turn and multilingual instructions following.arXiv preprint arXiv:2410.15553, 2024

    arXiv

  19. [27]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  20. [28]

    Pretrained transformers for text ranking: Bert and beyond

    Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. Pretrained transformers for text ranking: Bert and beyond. InProceedings of the 14th ACM International Conference on web search and data mining, pages 1154–1156, 2021

    2021

  21. [29]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  22. [30]

    Long-context llms struggle with long in-context learning.arXiv preprint arXiv:2404.02060, 2024

    Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning.arXiv preprint arXiv:2404.02060, 2024

    2024 arXiv

  23. [31]

    Bench: Extending long context evaluation beyond 100k tokens

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. Bench: Extending long context evaluation beyond 100k tokens. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (...

    2024

  24. [32]

    Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.arXiv preprint arXiv:2009.14794, 2020

    2009 arXiv

  25. [33]

    Beyond the limits: A survey of techniques to extend the context length in large language models.arXiv preprint arXiv:2402.02244, 2024

    Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Ar- maghan Eshaghi. Beyond the limits: A survey of techniques to extend the context length in large language models.arXiv preprint arXiv:2402.02244, 2024

    2024 arXiv

  26. [34]

    Longrope: Extending llm context window beyond 2 million tokens.arXiv preprint arXiv:2402.13753, 2024

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens.arXiv preprint arXiv:2402.13753, 2024

    2024 arXiv

  27. [35]

    Stateful memory-augmented transformers for dialogue modeling

    Qingyang Wu and Zhou Yu. Stateful memory-augmented transformers for dialogue modeling. arXiv preprint arXiv:2209.07634, 2022. 12

    2022 arXiv

  28. [36]

    Augmenting language models with long-term memory.Advances in Neural Information Processing Systems, 36, 2024

    Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory.Advances in Neural Information Processing Systems, 36, 2024

    2024

  29. [37]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing...

    2021 doi

  30. [38]

    Balancing accuracy and efficiency in multi-turn intent classification for llm-powered dialog systems in production.arXiv preprint arXiv:2411.12307, 2024

    Junhua Liu, Yong Keat Tan, Bin Fu, and Kwan Hui Lim. Balancing accuracy and efficiency in multi-turn intent classification for llm-powered dialog systems in production.arXiv preprint arXiv:2411.12307, 2024

    2024 arXiv

  31. [39]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  32. [40]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, and others. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv e-p...

    2024 doi arXiv

  33. [41]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  34. [42]

    Xing, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2024. URL https://arxiv.org/abs/...

    2024 arXiv

  35. [43]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  36. [44]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.ACM computing surveys, 55(9):1–35, 2023

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.ACM computing surveys, 55(9):1–35, 2023

    2023

  37. [45]

    Project Gutenberg, 2001

    Herman Melville.Moby-Dick; or, The Whale. Project Gutenberg, 2001. URL https://www. gutenberg.org/ebooks/2701. Accessed: 2025-02-11

    2001

  38. [46]

    Project Gutenberg, 1993

    Mary Shelley.Frankenstein; or, The Modern Prometheus. Project Gutenberg, 1993. URL https://www.gutenberg.org/ebooks/84. Accessed: 2025-02-11. 13 Figure 4: An illustration of varying context scenario in interactive settings where LLMs respond differently to the same target quer...

    1993

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.