Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Asking Again and Again: Exploring LLM Robustness to Repeated Questions

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Repeating a question 1, 3, or 5 times within a prompt does not significantly change the reading-comprehension accuracy of large language models, across five models, three datasets, and four prompt settings.

desk verdict A clean, honest null-result study on question repetition that overreaches its pooled statistics when claiming 'no meaningful impact' per setting. read the letter →

arxiv 2412.07923 v3 pith:WBR4YGKP submitted 2024-12-10 cs.CL

Sagi Shaier , Mario Sanz-Guerrero , Katharina von der Wense This is my paper · ORCID
classification cs.CL
keywords largelanguagemodelsquestionrepetitionpromptengineeringreadingcomprehensionLLMrobustnessstatisticalsignificanceFriedmantestdesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether repeating a question inside a single prompt helps or hurts large language models. Across five models (GPT-4o-mini, DeepSeek-V3, Llama-3.1 8B, Mistral 7B, Phi-4 14B), three reading-comprehension datasets, and four prompt configurations, the authors compare accuracy when the question appears 1, 3, or 5 times. They find that repetition can move accuracy by up to 6% in specific cells, but a pooled Friedman test yields p = 0.70, meaning the overall effect is not statistically significant. The paper concludes that verbatim or paraphrased question repetition alone is not a reliable prompt-engineering lever for reading comprehension, and that the tested models are robust to redundant input structures.

What carries the argument

The key machinery is a controlled prompt-repetition experiment with four configurations: open-book (context before question), closed-book (no context), question-context-question (QCQ, question both before and after context), and paraphrasing (original question plus model-generated paraphrases appended to context). Repetition levels are Qx1, Qx3, and Qx5, and accuracy is measured by substring matching of the gold answer. The statistical workhorse is the Friedman test, a non-parametric repeated-measures test that ranks the three repetition conditions within each experimental block; here it produced a p-value of 0.70, which carries the conclusion that repetition has no significant aggregate effect.

What would settle it

Run the same 90,000-question experiment but perform separate Friedman tests or paired comparisons for each model × dataset × configuration cell (or at least for cells with large swings, such as Llama-3.1 closed-book HotPotQA, which rises from 0.24 to 0.30, or Mistral open-book HotPotQA, which falls from 0.51 to 0.48). If any cell shows a statistically significant difference across Qx1, Qx3, and Qx5 after multiple-comparison correction, the blanket 'no meaningful impact' claim would be false. A simpler check is a permutation test that respects block structure to see whether the observed 6% gains exceed what chance would produce, or a larger-sample replication of one high-swing cell to determine whether the difference is real.

Watch

Extended reading notes

Core claim

The central claim is that question repetition within a prompt neither significantly improves nor significantly degrades LLM performance across all models, datasets, and settings tested. The evidence comes from 90,000 questions (500 sampled per dataset × 3 repetition levels × 4 configurations × 3 datasets × 5 models), with accuracy measured by substring matching. A Shapiro-Wilk test showed non-normality, so the authors used the non-parametric Friedman test on the pooled accuracy scores across repetition levels; the test statistic was 0.7118 with p = 0.70. Individual configurations show contrasting movements—for example, Llama-3.1 closed-book HotPotQA rises from 0.24 to 0.30 from Qx1 to Qx5, while Mistral open-book HotPotQA falls from 0.51 to 0.48—but the aggregate analysis treats these as noise. The authors interpret this as evidence that LLMs process the question effectively regardless of how many times it is repeated, and that repetition does not encourage the model to focus more on the repeated information.

Load-bearing premise

The blanket conclusion rests on the assumption that pooling all models, datasets, and configurations into one Friedman test is a valid way to detect repetition effects, which requires that meaningful repetition effects, if any, are consistent enough in direction to show up in the global rank comparison.

Editorial extensions

If this is right

  • Question repetition is not a dependable prompt-engineering technique for reading-comprehension tasks; practitioners should not expect verbatim or paraphrased repetition to improve accuracy.
  • The tested LLMs are robust to redundant phrasing, suggesting that simply showing a question multiple times does not strengthen the model's attention to it.
  • The null result contrasts with prior work showing benefits from instructing models to restate questions, so the mechanism behind such gains is not mere repeated presentation.
  • The conclusion holds across a wide range of model sizes (7B to 685B) and context conditions, indicating a general robustness rather than a quirk of a single model.
  • Any true repetition effects that exist are likely small and setting-specific, and would require larger samples or per-condition analyses to detect reliably.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single pooled Friedman test may cancel out opposing effects: if repetition helps smaller models in closed-book settings but hurts others, the aggregate null result could hide real, heterogeneous behavior that a per-cell analysis would reveal.
  • The 500-question subsamples may be underpowered for detecting small but genuine accuracy differences; a power analysis could show whether the experiment could have detected a 1–2% effect, and if not, the 'no meaningful impact' conclusion is weaker than it appears.
  • The paper's own limitation about causal-only models suggests an untested extension: masked or encoder-only language models might show a different sensitivity to question repetition, since they process input dependencies differently.
  • The interaction between repetition and explicit instructions (e.g., 'focus on the repeated question') is not tested here; combining repetition with such directives could still produce a measurable effect, contrary to the paper's broad null framing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether repeating a question 1, 3, or 5 times within a single prompt affects the accuracy of five LLMs (GPT-4o-mini, DeepSeek-V3, Llama-3.1 8B, Mistral 7B, Phi-4 14B) on SQuAD, HotPotQA, and Natural Questions, under open-book, closed-book, question-context-question, and paraphrasing configurations. Accuracy is measured by substring matching on 500 sampled questions per condition, for a total of 90,000 prompted evaluations. A Friedman test across the pooled condition blocks yields p=0.70, and the authors conclude that question repetition neither significantly improves nor significantly degrades performance across all models, datasets, and settings. The paper also reports descriptive per-configuration trends, including a 6% accuracy gain for Llama-3.1 on closed-book HotPotQA.

Significance. This is a carefully executed empirical study with a clearly stated null result. The scope is substantial: five recent LLMs, three QA benchmarks, four prompt configurations, and 500 questions per condition. The main positive finding—that literal question repetition produces at most small accuracy changes on reading-comprehension tasks—is a useful data point for prompt engineering, and the paper correctly contrasts with EchoPrompt's restatement-based gains. However, the statistical support for the strong universal claim is currently inadequate, as detailed in the major comments.

major comments (3)
  1. [§3.3, Appendix C.1, Table 1] The universal negative claim—that question repetition 'neither significantly improves nor significantly degrades the performance of LLMs across all models, datasets, and settings'—is supported only by a single Friedman test on pooled block ranks. This test is not sensitive to effects that are large in a few cells but opposite in direction across the 60 model×configuration×dataset blocks. Table 1 shows several 3–6 percentage point changes (e.g., Llama-3.1 closed-book HotPotQA Qx1=0.24 vs. Qx5=0.30; Mistral 7B open-book HotPotQA Qx1=0.51 vs. Qx5=0.48). With n=500 per cell, a 6-point difference is roughly two standard errors and may be statistically significant for that cell. The paper reports no per-cell significance tests, confidence intervals, effect sizes, or power analysis. Interpreting p=0.70 as 'no meaningful impact' therefore conflates absence of evidence with evidence of absence. I request per-cell paired tests (e.g., McNemar) with multiple-comparison correction, plus a power analysis showing the detectable effect size at n=500.
  2. [§2.1, Appendix A.4] The Paraphrasing configuration is pooled together with the identical-repetition conditions in the Friedman test, but it is a different manipulation: it introduces model-generated paraphrases rather than literal repetition. If repetition effects are masked by paraphrase noise, pooling reduces power. The paper should either analyze Paraphrasing separately or provide a justification for pooling. Also, the definition of repetition levels for Paraphrasing is unclear: Qx3 appears to correspond to one original plus two paraphrases, not three repetitions, so the column labels are misleading.
  3. [Abstract, §3.1] The abstract states that repetition 'can increase models’ accuracy by up to 6%' while §3.3 concludes there is 'no meaningful impact.' These statements are not contradictory only if the 6% increase is shown to be statistically indistinguishable from noise. The paper does not provide a confidence interval for that 6% cell (Llama-3.1 closed-book HotPotQA), so the reader cannot evaluate whether the largest observed effect is real. Please report the uncertainty around the largest per-cell differences, or soften the universal claim to 'no consistent statistically significant effect across the pooled analysis.'
minor comments (5)
  1. [§3.1, §3.2] There are subject-verb agreement errors, e.g., 'performance remain stable' (§3.1) and 'NQ show the smallest' (§3.2); the manuscript should be proofread for such grammatical issues.
  2. [§2.3] The phrase 'total of 90,000 questions' is ambiguous; since the same 500 questions are reused across all 180 settings, the correct description is 90,000 prompted evaluations, not 90,000 unique questions.
  3. [Appendix A] The prompt templates are only described with placeholders; for reproducibility, please provide the exact full prompts for Qx1, Qx3, and Qx5 in each configuration.
  4. [General] No code, data, or model-output release is mentioned; given the use of API models, releasing the prompt templates, sampled question IDs, and inference configuration would substantially improve reproducibility.
  5. [Figures] Figures 2–5 are referenced in the text but appear after the appendix; please ensure all figures are called out in numerical order in the final version and that captions are self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical benchmark evaluation whose conclusion rests on direct model outputs and external datasets, not on fitted parameters or self-referential derivation.

full rationale

The paper's central claim, that question repetition within a prompt neither significantly improves nor significantly degrades LLM performance, is an empirical finding based on measured accuracies of five models on three external reading-comprehension datasets (SQuAD, HotPotQA, Natural Questions). These accuracies are direct observations of model outputs under different prompt variations, not quantities derived from the paper's own assumptions or fitted to the outcome being claimed. The statistical test in Appendix C.1 (Friedman test with p = 0.70) summarizes the observed accuracy data; it is not a parameter fitted to a subset of that data and then used to predict the same data. The only self-citations appear in background choices: citing Shaier et al. (2024a) for placing context before the question in the open-book setting, and for using long answers as context in Natural Questions. These prompt-construction choices are antecedent to the experiment and do not determine whether repetition helps or hurts; they are not load-bearing for the null result. No uniqueness theorem, imported ansatz, or renaming of a known result is invoked. The limitation that the pooled Friedman test may be underpowered, or that per-cell differences in Table 1 might be heterogeneous, is a legitimate statistical-correctness concern, but it is not a form of circularity. The derivation chain is therefore self-contained with respect to the paper's own conclusions.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

This empirical study does not fit any parameters and introduces no new theoretical entities. Its central claim depends on the validity of the substring-match accuracy metric, the representativeness of the 500-question samples, the suitability of a pooled Friedman test, the stability of single-run model outputs, and the quality of model-generated paraphrases. These are domain assumptions rather than mathematical axioms.

assumptions (5)
  • domain assumption Substring matching is a valid measure of model accuracy.
    Used in Section 2.2 to define accuracy. If the metric is too lenient or too strict, the reported accuracies and effect sizes may not reflect true performance.
  • domain assumption The 500-question sample per dataset is representative of the full dataset.
    Section 2.3 samples 500 questions per dataset for cost reasons but does not describe the sampling procedure. If the sample is not random or stratified, results may not generalize.
  • domain assumption The global Friedman test on pooled data is appropriate for detecting repetition effects.
    Appendix C.1 applies a single Friedman test across all models, datasets, and configurations, assuming repetition effects are consistent across blocks. Pooling heterogeneous effects could mask real differences.
  • domain assumption A single forward pass per prompt yields a stable estimate of model accuracy.
    No temperature, seed, or multiple-run averaging is reported in Section 2.4. LLM sampling is stochastic, especially for API models, so point estimates in Table 1 could vary across runs.
  • domain assumption Paraphrased questions generated by the model are a valid form of the repetition manipulation.
    Appendix A.4 specifies the paraphrasing prompt but not which model generated the paraphrases or the decoding parameters. Low-quality paraphrases would make this condition a test of paraphrase quality rather than repetition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Asking Again and Again: Exploring LLM Robustness to Repeated Questions." pith.science (2026). https://pith.science/paper/WBR4YGKP

@misc{pith2026241207923,
  author       = {Pith},
  title        = {Pith review of: Asking Again and Again: Exploring LLM Robustness to Repeated Questions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WBR4YGKP}},
  note         = {Machine review of arXiv:2412.07923}
}
abstract

This study investigates whether repeating questions within prompts influences the performance of large language models (LLMs). We hypothesize that reiterating a question within a single prompt might enhance the model's focus on key elements of the query. We evaluate five recent LLMs -- including GPT-4o-mini, DeepSeek-V3, and smaller open-source models -- on three reading comprehension datasets under different prompt settings, varying question repetition levels (1, 3, or 5 times per prompt). Our results demonstrate that question repetition can increase models' accuracy by up to $6\%$. However, across all models, settings, and datasets, we do not find the result statistically significant. These findings provide insights into prompt design and LLM behavior, suggesting that repetition alone does not significantly impact output quality.

Figures

Figures reproduced from arXiv: 2412.07923 by the authors.

Figure 1
Figure 1. Overview of our study on question repetition [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Average accuracy by configuration and repeti [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Average accuracy by model and repetition [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Average accuracy by configuration and model [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Average accuracy by dataset and model across [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Repeating the question on both sides of the image (question echoing) closes the question-first accuracy gap in five open VLMs and beats standard single-pass orderings on several VQA benchmarks.

Reference graph

Works this paper leans on

34 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu,...

    arXiv 2024

  4. [4]

    Dmitriy Alexandrov, Anastasiia Zakharova, and Nikolay Butakov. 2023. https://doi.org/10.1109/ICSC56153.2023.00012 Does noise really matter? investigation into the influence of noisy labels on bert-based question answering system . In 2023 IEEE 17th International Conference on Semantic Computing (ICSC), pages 33--40

    arXiv 2023

  5. [5]

    Anthropic. 2024. https://assets.anthropic.com/m/61e7d27f8c8f5919/original/Claude-3-Model-Card.pdf The claude 3 model family: Opus, sonnet, haiku . Technical Report

    work page 2024

  6. [6]

    Minwook Bae and Hyounghun Kim. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1046 Collective critics for creative story generation . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18784--18819, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.emnlp-main.1046 2024

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  8. [8]

    Yu Cao, Dianqi Li, Meng Fang, Tianyi Zhou, Jun Gao, Yibing Zhan, and Dacheng Tao. 2022. https://arxiv.org/abs/2210.15221 Tasa: Deceiving question answering models by twin answer sentences attack . Preprint, arXiv:2210.15221

    arXiv 2022

Show all 34 references
  1. [9]

    Hung-Ting Chen, Michael Zhang, and Eunsol Choi. 2022. https://aclanthology.org/2022.emnlp-main.146 Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence . In Proceedings of the 2022 Conference on Empirical Methods in Nat...

    2022

  2. [10]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vino...

    2022 arXiv

  3. [11]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2024 arXiv

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The Llama 3 Herd of Models . Preprint, arXiv:2407.21783

    2024 arXiv

  5. [13]

    Giwon Hong, Jeonghwan Kim, Junmo Kang, Sung-Hyon Myaeng, and Joyce Jiyoung Whang. 2023. https://arxiv.org/abs/2305.01579 Discern and answer: Mitigating the impact of misinformation in retrieval-augmented models with discriminators . Preprint, arXiv:2305.01579

    2023 arXiv

  6. [14]

    Robin Jia and Percy Liang. 2017. https://doi.org/10.18653/v1/D17-1215 Adversarial examples for evaluating reading comprehension systems . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021--2031, Copenhagen, Denmark. Associati...

    2017 doi

  7. [15]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. https://arxiv.org/abs/2310.06825 Mistral 7b . Preprint, arXiv:2310.06825

    2023 arXiv

  8. [16]

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. https://arxiv.org/abs/2211.08411 Large language models struggle to learn long-tail knowledge . Preprint, arXiv:2211.08411

    2023 arXiv

  9. [17]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  10. [18]

    Sharon Levy, Tahilin Sanchez Karver, William Adler, Michelle R Kaufman, and Mark Dredze. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.332 Evaluating biases in context-dependent sexual and reproductive health questions . In Findings of the Association for Computational...

    2024 doi

  11. [19]

    Gonzalez, Ion Stoica, Xuezhe Ma, , and Hao Zhang

    Dacheng Li, Rulin Shao*, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, , and Hao Zhang. 2023. https://lmsys.org/blog/2023-06-29-longchat How long can open-source llms truly promise on context length?

    2023

  12. [20]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. https://arxiv.org/abs/2307.03172 Lost in the middle: How language models use long contexts . Preprint, arXiv:2307.03172

    2023 arXiv

  13. [21]

    Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.565 Entity-based knowledge conflicts in question answering . In Proceedings of the 2021 Conference on Empirical Methods in Natural L...

    2021 doi

  14. [22]

    Yan Ma, Yu Qiao, and Pengfei Liu. 2024. https://doi.org/10.18653/v1/2024.acl-long.117 M o PS : Modular story premise synthesis for open-ended automatic story generation . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2024 doi

  15. [23]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.546 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . In Proceedings of the 6...

    2023 doi

  16. [24]

    Rajasekhar Reddy Mekala, Yasaman Razeghi, and Sameer Singh. 2024. https://arxiv.org/abs/2309.10687 Echoprompt: Instructing the model to rephrase queries for improved in-context learning . Preprint, arXiv:2309.10687

    2024 arXiv

  17. [25]

    Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2022. https://doi.org/10.48550/ARXIV.2211.05655 Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering . arXiv preprint

    2022 doi arXiv

  18. [26]

    OpenAI. 2023 a . https://openai.com/blog/chatgpt/ Chatgpt: Optimizing language models for dialogue

    2023

  19. [27]

    OpenAI. 2023 b . https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2023 arXiv

  20. [28]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  21. [29]

    Sagi Shaier, Kevin Bennett, Lawrence Hunter, and Katharina Kann. 2023. https://aclanthology.org/2023.ijcnlp-main.36 Emerging challenges in personalized medicine: Assessing demographic effects on biomedical question answering systems . In Proceedings of the 13th International J...

    2023

  22. [30]

    Sagi Shaier, Lawrence Hunter, and Katharina von der Wense. 2024 a . https://aclanthology.org/2024.findings-acl.491 It is not about what you say, it is about how you say it: A surprisingly simple approach for improving reading comprehension . In Findings of the Association for ...

    2024

  23. [31]

    Sagi Shaier, Lawrence Hunter, and Katharina Wense. 2024 b . https://aclanthology.org/2024.eacl-long.47 Desiderata for the context use of question answering systems . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics ...

    2024

  24. [32]

    Sagi Shaier, Ari Kobren, and Philip V. Ogren. 2024 c . https://doi.org/10.18653/v1/2024.emnlp-main.956 Adaptive question answering: Enhancing language model proficiency for addressing knowledge conflicts with source citations . In Proceedings of the 2024 Conference on Empirica...

    2024 doi

  25. [33]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  26. [34]

    Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. https://arxiv.org/abs/2303.11315 Context-faithful prompting for large language models . Preprint, arXiv:2303.11315

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.