Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QA

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Question evergreen-ness is learnable and language-independent: a small classifier reaches 0.91 F1 and improves self-knowledge, dataset curation, and retrieval analysis.

desk verdict EverGreenQA and EG-E5 are real contributions; the application sections overclaim and need substantive revision. read the letter →

arxiv 2505.21115 v1 pith:HG24SYDS submitted 2025-05-27 cs.CL

Sergey Pletenev , Maria Marina , Nikolay Ivanov , Daria Galimzianova , Nikita Krayko , Mikhail Salnikov , Vasily Konovalov , Alexander Panchenko
show 1 more author
Viktor Moskvoretskii
This is my paper
classification cs.CL
keywords questiontemporalityevergreenquestionsmultilingualQAdatasetself-knowledgeestimationuncertaintyquantificationretrieval-augmentedgenerationcurationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many question-answering failures come from questions whose answers change over time, yet the field has no reliable way to tell those apart from stable, or 'evergreen,' questions. This paper introduces EverGreenQA, a human-labeled multilingual dataset of 4,757 questions in seven languages, and shows that the evergreen/mutable distinction is learnable: a fine-tuned multilingual E5 encoder, EG-E5, classifies it at about 0.91 weighted F1, matching or beating few-shot LLMs. It then shows the practical payoff: the predicted evergreen probability improves self-knowledge estimation (16 of 18 settings), exposes mutable questions lurking in standard QA benchmarks, and predicts GPT-4o's retrieval decisions better than any uncertainty measure tested.

What carries the argument

The central object is the binary label of question evergreen-ness, defined as whether the answer has remained stable for 20–30 years, as judged by trained linguists. The mechanism that carries the argument is EG-E5, a single multilingual E5-Large encoder fine-tuned on EverGreenQA, which outputs an 'evergreen probability' for any question in seven languages. This probability is then used as a feature in self-knowledge classifiers, as a filter for QA benchmarks, and as a correlate of retrieval behavior; the paper also shows that uncertainty-based signals (perplexity, token entropy) carry only weak implicit information about the same property.

What would settle it

Take a random sample of EverGreenQA in each language, have independent annotators relabel it without using GPT-4.1 translations and with a different stability rule (e.g., 5 years instead of 30), and retrain EG-E5 on the new labels; if cross-language agreement collapses or the new classifier fails to reproduce the self-knowledge improvements, the claim that evergreen-ness is a language-independent learnable property would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that 'evergreen-ness'—whether a question's correct answer remains stable for roughly 20–30 years—is a real, cross-lingual property of questions rather than a surface artifact, and that it can be captured by a lightweight classifier trained on human labels. The authors construct EverGreenQA with linguist-assigned labels, translate it into six languages, and train EG-E5, a single multilingual E5-Large model, which reaches 0.897–0.913 weighted F1 across all seven languages. They further claim that this signal strengthens self-knowledge identification: in 16 of 18 evaluations the best result uses evergreen probability alone or combined with uncertainty estimates, and it improves calibration. The evergreen label also separates questions that models answer correctly from those they miss (accuracy gaps up to 40%), and it is the strongest predictor of when GPT-4o triggers a retrieval call.

Load-bearing premise

The load-bearing premise is that the linguist-assigned evergreen/mutable labels, made under a 'stable for 20–30 years' rule, remain valid when questions are translated with GPT-4.1 into six languages and when the same property is applied to external QA datasets; if the labels shift under translation or annotation, the high F1 and the self-knowledge gains could reflect label artifacts rather than a true temporal property.

Editorial extensions

If this is right

  • Evergreen probability can be added to standard uncertainty estimators to improve self-knowledge estimation, matching or beating the best result in 16 of 18 evaluation settings and often improving calibration.
  • QA benchmarks contain a nontrivial share of mutable questions, up to 18% in Natural Questions and 10% on average, so filtering with an evergreen classifier can make evaluations fairer and more temporally stable.
  • Models answer evergreen questions more reliably, with relative accuracy gaps up to 40% in multi-hop tasks, while retrieval augmentation helps more on mutable questions, suggesting evergreen scores could guide when to retrieve.
  • Evergreen-ness is the strongest tested predictor of GPT-4o's black-box retrieval decisions, more than twice as informative as uncertainty signals, indicating that retrieval policies are closely tied to question temporality.
  • EG-E5 is a lightweight single encoder that maintains high F1 across Latin and non-Latin scripts, making evergreen classification practical in low-compute settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The evergreen probability could serve as a cheap pre-filter for retrieval-augmented generation, routing mutable questions to search and answering evergreen ones from parametric memory, which would cut latency and retrieval cost.
  • Because the paper defines evergreen-ness with a fixed 20–30 year stability rule, a natural extension is to predict a continuous stability horizon or a domain-specific change rate, which may fix the reported errors on superlatives and living persons.
  • The strong correlation with GPT-4o retrieval may mean that retrieval policies are trained to recognize potentially changing facts, so evergreen classification doubles as an interpretability tool for black-box retrieval.
  • A testable follow-up is to check whether the evergreen probability also predicts how quickly a model's answer becomes outdated, turning a binary label into a temporal-drift forecasting signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces EverGreenQA, a human-labeled multilingual dataset of 4,757 questions in seven languages with binary evergreen/mutable labels, and trains EG-E5, a multilingual E5-Large classifier reporting roughly 0.91 weighted F1. It benchmarks twelve LLMs with verbalized prompting and uncertainty signals, and tests the practical value of evergreen probability in self-knowledge estimation, QA dataset filtering, and explaining GPT-4o retrieval behavior. The central claim is that evergreen-ness is a learnable, largely language-independent property that can improve trustworthy QA at low cost.

Significance. If the label property is stable and transferable, the contribution is useful: EverGreenQA is the first multilingual evergreen-labeled QA dataset with a train/test split, the EG-E5 classifier is lightweight and released with the data, and the three applications address real problems in trustworthy QA. The paper also provides a broad benchmark of twelve LLMs and a useful qualitative error analysis. However, the practical claims currently exceed the evidence, because the reliability of the labels and their transfer across languages and datasets are not established, and several Table 4 results contradict the narrative of consistent improvement.

major comments (3)
  1. [5.5, Table 4] The claim that 'evergreen probability is a strong signal' and that the best results are achieved in 16 out of 18 evaluations is not supported by the table. For example, LexicalSimilarity AUROC on NQ drops from 0.61 to 0.59 with +EG, MaxTokenEntropy AUROC drops from 0.61 to 0.56, and 2WikiMultihopQA PRR drops from 0.71 to 0.64. The evergreen feature alone has AUROC near 0.50 on NQ, SQuAD, HotpotQA, and MuSiQue. No confidence intervals or standard deviations are reported even though Appendix D states that three random seeds were used, and class prevalences are not given, making the high AUPRC values hard to interpret. Please report variance, per-dataset mutable prevalence, and significance tests, and revise the 'consistent improvements' wording.
  2. [3, Appendix F.2] The core assumption that evergreen/mutable is a stable, transferable property is not validated. Labels are assigned by one team of linguists using a 20-30 year heuristic with no inter-annotator agreement reported. Translation validation in Section 3 checks only translation fidelity on 100 questions per language, not whether the evergreen/mutable distinction survives translation. The high multilingual F1 of EG-E5 may therefore reflect annotator-specific labeling style rather than a linguistic universal. Please report kappa or a re-annotation study, have target-language annotators label translated items, and validate EG-E5 against human evergreen labels on at least one external English QA dataset.
  3. [6.1, Table 6] The dataset-filtering conclusions rest on unvalidated classifier predictions. The mutable-question proportions in Table 6 (e.g., 18% for NQ) appear to be EG-E5 predictions on external datasets, but no human validation is provided for those datasets, and the 2WikiMultihopQA row is based on about 0.5 mutable questions in a 500-question subset, making the reported -70% RAG gain meaningless. Report raw counts of mutable/evergreen predictions, human-verify a sample on external datasets, and base aggregate conclusions on datasets with sufficient mutable examples.
minor comments (5)
  1. [Limitations] The Limitations section says the dataset size is 3,278 examples and that 14 models were evaluated, while Section 3 and Table 2 state 4,757 questions and 12 models; these numbers should be reconciled.
  2. [Table 6] The column header '% EG Mut' is ambiguous; rename it to '% mutable' and clarify in the caption how the proportion was computed.
  3. [5.5] The '16 out of 18 evaluations' statistic should state exactly which dataset-metric cells are being counted, since Table 4 contains five methods, one evergreen feature, and three metrics across six datasets.
  4. [Table 2] The random baseline of 0.637 is reported without the class distribution; report the proportion of evergreen examples in the test split so the baseline is interpretable.
  5. [Appendix F.2] The '20-30 years of stability' criterion is central to the label definition and should be introduced in Section 3 rather than only in the appendix.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: EG-E5 is evaluated on a held-out test split and the evergreen feature is a fixed signal on external QA data, not a fitted predictor of the target.

full rationale

The paper's central claims are not circular by construction. EverGreenQA is split into 3,487 training and 1,270 testing examples, and EG-E5's reported 0.91 weighted F1 is measured on that held-out test split, which is standard supervised evaluation rather than a self-fulfilling prediction. In the self-knowledge experiments (Section 5), the evergreen probability is produced by the already-trained EG-E5 and then used as a fixed additional feature for classifiers trained to predict answer correctness on six external QA datasets; the evergreen feature is not fitted to the self-knowledge labels, so the reported improvement is an empirical transfer result rather than a fitted input renamed as a prediction. The dataset filtering and RAG analyses similarly apply the classifier to new data and compare accuracy or RAG gain across predicted classes; even if those comparisons are partly expected from the definition of mutability, they are not derived from the classifier's training target by construction. The paper does contain self-citations, notably Moskvoretskii et al. (2025) for self-knowledge and uncertainty measures, but the same claims are also supported by external references such as Yin et al. (2023), Vashurin et al. (2024), and Fadeeva et al. (2024), so the self-citations are not load-bearing. Concerns about label reliability, lack of inter-annotator agreement, and limited translation validation are validity threats, not circularity: they question whether the labels capture a real property, but they do not show that any result reduces to its inputs by definition or by fitting.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on human labeling judgments, machine translation, feature transfer, and observational measurement. The 20-30 year stability rule is a hand-chosen parameter that propagates into all labels. No physical entities or new theoretical constructs are introduced.

free parameters (2)
  • Evergreen stability horizon = 20-30 years (hand-chosen)
    Annotators treat facts unchanged for 20-30 years as established (Appendix F.2); this threshold determines every label in EverGreenQA and thus all downstream results.
  • Synthetic evergreen examples added = 1,449
    Generated by GPT-4.1 and manually validated to balance classes (Section 3); the exact number affects training distribution and classifier behavior.
assumptions (5)
  • domain assumption Evergreen-ness is a binary, well-defined property of questions with a single correct label.
    The whole dataset construction, classifier training, and evaluation treat evergreen vs mutable as exhaustive and disjoint (Section 3, Table 2).
  • domain assumption GPT-4.1 machine translation preserves both the semantics and the evergreen/mutable label of each question.
    Dataset translation relies on GPT-4.1 and only 100 samples per language were human-validated (Section 3, Dataset Translation and Validation).
  • domain assumption Perplexity and mean token entropy of a model's generated answer reflect its internal knowledge uncertainty.
    Section 4.2 uses these signals as implicit measures of evergreen awareness; if they do not measure knowledge, the correlation analysis is uninterpretable.
  • domain assumption EG-E5's evergreen probability transfers from EverGreenQA to NQ, SQuAD, TriviaQA, HotpotQA, MuSiQue, and 2WikiMultihopQA without distribution shift.
    Section 5.4 applies EG-E5 to six QA datasets and uses its output as a self-knowledge feature; no domain adaptation or validation on those datasets is reported.
  • domain assumption Observing GPT-4o's web interface and recording whether it triggers retrieval is a reliable measure of its internal retrieval policy.
    Section 7 uses manual observation of the web interface; no controlled API, no inter-annotator agreement, and no protocol details are provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QA." pith.science (2026). https://pith.science/paper/HG24SYDS

@misc{pith2026250521115,
  author       = {Pith},
  title        = {Pith review of: Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HG24SYDS}},
  note         = {Machine review of arXiv:2505.21115}
}
read the original abstract

Large Language Models (LLMs) often hallucinate in question answering (QA) tasks. A key yet underexplored factor contributing to this is the temporality of questions -- whether they are evergreen (answers remain stable over time) or mutable (answers change). In this work, we introduce EverGreenQA, the first multilingual QA dataset with evergreen labels, supporting both evaluation and training. Using EverGreenQA, we benchmark 12 modern LLMs to assess whether they encode question temporality explicitly (via verbalized judgments) or implicitly (via uncertainty signals). We also train EG-E5, a lightweight multilingual classifier that achieves SoTA performance on this task. Finally, we demonstrate the practical utility of evergreen classification across three applications: improving self-knowledge estimation, filtering QA datasets, and explaining GPT-4o retrieval behavior.

Figures

Figures reproduced from arXiv: 2505.21115 by the authors.

Figure 1
Figure 1. Some questions have answers that stay the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 4 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A hi...

    arXiv 2024

  2. [2]

    Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Ga \" e l Varoquaux. 2013. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Wor...

    work page 2013

  3. [3]

    Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/1f0e3dad99908345f7439f8ffabdffc4-Abstract-round2.html A dataset for answering time-sensitive questions . In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021...

    2021

  4. [4]

    Qinyuan Cheng, Xiaonan Li, Shimin Li, Qin Zhu, Zhangyue Yin, Yunfan Shao, Linyang Li, Tianxiang Sun, Hang Yan, and Xipeng Qiu. 2024. https://arxiv.org/abs/2406.12534 Unified active retrieval for retrieval augmented generation . Preprint, arXiv:2406.12534

    arXiv 2024

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    arXiv 2019

  6. [6]

    Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng. 2024. https://doi.org/10.48550/ARXIV.2402.10612 Retrieve only when it needs: Adaptive retrieval augmentation for hallucination mitigation in large language models . CoRR, abs/2402.10612

    doi:10.48550/arxiv.2402.10612 2024

  7. [7]

    Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2023. Shifting attention to relevance: Towards the uncertainty estimation of large language models. arXiv preprint arXiv:2307.01379

    arXiv 2023

  8. [8]

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, and 1 others. 2024. Fact-checking the output of large language models via token-level uncertainty quantification. arXiv preprint arXiv:2403.04696

    arXiv 2024

Show all 44 references
  1. [9]

    Constanza Fierro, Nicolas Garneau, Emanuele Bugliarello, Yova Kementchedjhieva, and Anders S gaard. 2024. https://doi.org/10.18653/V1/2024.NAACL-SHORT.67 Mulan: A study of fact mutability in language models . In Proceedings of the 2024 Conference of the North American Chapter ...

    2024 doi

  2. [10]

    Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Fr \'e d \'e ric Blain, Francisco Guzm \'a n, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computation...

    2020

  3. [11]

    Xiou Ge, Ali Mousavi, Edouard Grave, Armand Joulin, Kun Qian, Benjamin Han, Mostafa Arefiyan, and Yunyao Li. 2024. https://doi.org/10.18653/v1/2024.acl-short.53 Time sensitive knowledge editing through efficient finetuning . In Proceedings of the 62nd Annual Meeting of the Ass...

    2024 doi

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  5. [13]

    John T Hancock and Taghi M Khoshgoftaar. 2020. Catboost for big data: an interdisciplinary review. Journal of big data, 7(1):94

    2020

  6. [14]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. https://arxiv.org/abs/2111.09543 Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing . Preprint, arXiv:2111.09543

    2023 arXiv

  7. [15]

    Xanh Ho, Anh - Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/V1/2020.COLING-MAIN.580 Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational ...

    2020 doi

  8. [16]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and o...

    2025 doi

  9. [17]

    Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. 2024. https://doi.org/10.18653/V1/2024.NAACL-LONG.389 Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity . In Proceedings of the 2024 Conference of the Nor...

    2024 doi

  10. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  11. [19]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/V1/P17-1147 Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  12. [20]

    Smith, Yejin Choi, and Kentaro Inui

    Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2024. https://arxiv.org/abs/2207.13332 Realtime qa: What's the answer right now? Preprint, arXiv:2207.13332

    2024 arXiv

  13. [22]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187

    2023 arXiv

  14. [23]

    Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien De Masson D'Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-Mcmahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. https://proceedings.mlr.press/v...

    2022

  15. [24]

    Viktor Moskvoretskii, Maria Lysyuk, Mikhail Salnikov, Nikolay Ivanov, Sergey Pletenev, Daria Galimzianova, Nikita Krayko, Vasily Konovalov, Irina Nikishina, and Alexander Panchenko. 2025. Adaptive retrieval without self-knowledge? bringing uncertainty back home. arXiv preprint...

    2025 arXiv

  16. [25]

    Sergey Pletenev, Maria Marina, Daniil Moskovskiy, Vasily Konovalov, Pavel Braslavski, Alexander Panchenko, and Mikhail Salnikov. 2025. https://aclanthology.org/2025.findings-naacl.243/ How much knowledge can you pack into a L o RA adapter without harming LLM ? In Findings of t...

    2025

  17. [26]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/V1/D16-1264 Squad: 100, 000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, A...

    2016 doi

  18. [27]

    Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.702 DRAGIN: dynamic retrieval augmented generation based on the real-time information needs of large language models . In Proceedings of the 62nd Annual Meeting of t...

    2024 doi

  19. [28]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  20. [29]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  21. [30]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://doi.org/10.1162/TACL\_A\_00475 Musique: Multihop questions via single-hop question composition . Trans. Assoc. Comput. Linguistics, 10:539--554

    2022 doi

  22. [31]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. https://doi.org/10.18653/V1/2023.ACL-LONG.557 Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions . In Proceedings of the 61st Annual Meeting of th...

    2023 doi

  23. [32]

    Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2024. https://doi.o...

    2024 doi

  24. [33]

    Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadglign Ademtew, Yahya Hmaiti, Amandeep Kumar, Kartik Kuckreja, and 1 others. 2024. All languages matter: Evaluating lmms on culturally diverse 100 languages. arXiv pr...

    2024 arXiv

  25. [34]

    Artem Vazhentsev, Lyudmila Rvanova, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2025. Token-level density-based uncertainty quantification methods for eliciting truthfulness of large language models. arXiv preprint arXiv:2502.14427

    2025 arXiv

  26. [35]

    Akshaj Kumar Veldanda, Shi-Xiong Zhang, Anirban Das, Supriyo Chakraborty, Stephen Rawls, Sambit Sahu, and Milind Naphade. 2024. Llm surgery: Efficient knowledge unlearning and editing in large language models. arXiv e-prints, pages arXiv--2409

    2024

  27. [36]

    Wei, Jason Wei, Chris Tar, Yun - Hsuan Sung, Denny Zhou, Quoc V

    Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry W. Wei, Jason Wei, Chris Tar, Yun - Hsuan Sung, Denny Zhou, Quoc V. Le, and Thang Luong. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.813 Freshllms: Refreshing large language models with search engine augmentation . ...

    2024 doi

  28. [37]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. https://arxiv.org/abs/2402.05672 Multilingual e5 text embeddings: A technical report . Preprint, arXiv:2402.05672

    2024 arXiv

  29. [38]

    Yile Wang, Peng Li, Maosong Sun, and Yang Liu. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.691 Self-knowledge guided retrieval augmentation for large language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10,...

    2023 doi

  30. [40]

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024 b . Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368

    2024 arXiv

  31. [41]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/V1/D18-1259 Hotpotqa: A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  32. [42]

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don't know? arXiv preprint arXiv:2305.18153

    2023 arXiv

  33. [43]

    Michael J. Q. Zhang and Eunsol Choi. 2021. https://doi.org/10.18653/V1/2021.EMNLP-MAIN.586 Situatedqa: Incorporating extra-linguistic contexts into QA . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta...

    2021 doi

  34. [44]

    Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, and Noah A. Smith. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.892 Set the clock: Temporal alignment of pretrained language models . In Findings of the Association for Computational Linguistics, ACL 2024,...

    2024 doi

  35. [45]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.