Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Standard uncertainty scores often beat bespoke adaptive-retrieval pipelines on efficiency and self-knowledge while matching their answer accuracy.

desk verdict A useful, well-run benchmark whose headline 'comparable QA performance' is contradicted by the paper's own Table 1 on two of three multi-hop datasets. read the letter →

arxiv 2501.12835 v2 pith:REP4XXFJ submitted 2025-01-22 cs.CL cs.LG

Viktor Moskvoretskii , Maria Lysyuk , Mikhail Salnikov , Nikolay Ivanov , Sergey Pletenev , Daria Galimzianova , Nikita Krayko , Vasily Konovalov
show 2 more authors
Irina Nikishina Alexander Panchenko
This is my paper · ORCID
classification cs.CLcs.LG
keywords adaptiveretrievaluncertaintyestimationself-knowledgeretrieval-augmentedgenerationquestionansweringefficiencyout-of-distributiontransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Adaptive retrieval systems try to decide, question by question, whether a language model should consult external documents or answer from memory. The paper asks whether the elaborate pipelines built for this decision are actually better than plain uncertainty scores, and finds that they usually are not: across six QA datasets, 27 standard uncertainty-estimation techniques matched or outperformed eight recent adaptive-retrieval methods on answer accuracy while using far fewer language-model and retriever calls. The uncertainty methods also did better at self-knowledge, the ability to tell apart questions the model can answer from those it cannot. The practical upshot is that a simple confidence score plus a threshold may be the most efficient known way to gate retrieval, and that better self-knowledge, not better retrieval, is what still separates current systems from an oracle that retrieves only when necessary.

What carries the argument

The load-bearing mechanism is the self-knowledge score: each method is viewed as a function $f$ mapping an input question to a real number, where higher values indicate lower self-knowledge, and a classifier $C$ decides whether to retrieve. For uncertainty estimation methods this score comes from a standard uncertainty measure (mean or max token entropy, lexical similarity of sampled answers, Laplacian eigenvalue sums, or token-relevance reweighted entropy), and the classifier is trained once on a small set of questions and then applied at test time. Efficiency is counted as the number of language-model calls and retriever calls per question, and the paper contrasts every method against two anchors: 'Best UE', the top uncertainty score per dataset, and 'Ideal', an oracle that knows exactly when retrieval is needed, which bounds how much room remains for better self-knowledge.

What would settle it

Run the same 35-method comparison with the original released implementations and verification models for the baselines (notably Rowen and SeaKR), or repeat on a second base LLM; if the uncertainty methods cease to match the pipelines' QA accuracy while using fewer calls, the central claim is false.

Watch

Extended reading notes

Core claim

The paper's central discovery is that established uncertainty estimation (UE) techniques---simple scores computed from token probabilities, sampled-response consistency, or internal states---are often more useful for adaptive retrieval than the elaborate pipelines built for the task. Across 35 methods and 6 QA datasets, UE methods outperform the eight recent adaptive-retrieval baselines on single-hop questions and match them on multi-hop questions, while using two or fewer language-model calls and at most one retriever call per question, compared with pipelines that routinely make several to dozens of calls. UE methods also rank first or second on self-knowledge metrics, meaning their scores better separate questions the model can answer from those it cannot. No single method dominates: internal-state based SeaKR is strongest on single-hop self-knowledge, while the trainable reflexive classifier in AdaptiveRAG handles multi-hop questions better, and the paper reports that downstream accuracy and self-knowledge are only weakly correlated, so QA performance alone does not reveal whether a system knows when it needs external help.

Load-bearing premise

The comparison assumes that the reimplemented versions of the eight baseline pipelines, all run on LLaMA 3.1-8B and with Rowen's consistency-checking model swapped from Qwen-Max-0428 to Qwen 2.5-72B-Instruct, faithfully represent the original methods.

Editorial extensions

If this is right

  • A QA system can be made adaptive by thresholding a standard uncertainty score instead of running a multi-stage retrieval pipeline; on the six datasets tested this preserves answer accuracy while cutting language-model calls to two or fewer and retriever calls to at most one per question.
  • On single-hop questions the uncertainty-based gate is the best-performing adaptive strategy, and on multi-hop questions it matches the pipelines, so the complexity of bespoke adaptive retrieval buys little measured accuracy in this setup.
  • Because QA accuracy and self-knowledge are only weakly correlated (a moderate Spearman correlation of 0.65 between In-Accuracy and ROC-AUC ranks), methods should be evaluated on both axes; a system can score well on answers while routinely misjudging when it needs retrieval.
  • Out-of-domain transfer of uncertainty methods is stable for answer accuracy (typical loss under 4 percent), but self-knowledge accuracy degrades more and retriever calls increase, so deployment on a new distribution should budget for more retrieval.
  • The 'Ideal' oracle outperforms every real method on both accuracy and efficiency, indicating that better self-knowledge estimation, not better retrieval, is the remaining bottleneck for adaptive QA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A pragmatic takeaway the authors leave implicit: a production system could replace an adaptive-retrieval module with a single uncertainty score (e.g., mean entropy or lexical similarity) plus a threshold, and the paper's numbers suggest this would capture most of the accuracy benefit of the pipelines at a fraction of the compute.
  • The weak correlation between QA performance and self-knowledge implies that benchmark rankings of adaptive-retrieval methods are incomplete unless they also report calibration; a method could look strong on accuracy while systematically retrieving answers the model already knows.
  • Because internal-state methods showed the highest functional complexity and the largest out-of-domain self-knowledge drops, the simpler logit-based scores may generalize better as defaults; this is an inference from the paper's complexity and transfer analyses, not one of its stated conclusions.
  • A natural next experiment would be to combine a cheap uncertainty gate with a single retrieval pass and compare to the 'Ideal' bound; the paper's tables show an In-Accuracy gap between the best real method and the oracle on every dataset, suggesting headroom for better self-knowledge estimators.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a large-scale empirical comparison of 8 adaptive retrieval pipelines and 27 uncertainty estimation (UE) methods on 6 QA datasets, evaluating QA accuracy (In-Accuracy, EM, F1), self-knowledge identification (ROC-AUC, Spearman correlation, accuracy, over/underconfidence), and efficiency (number of LM calls and retriever calls). The central claim is that simple UE methods often match or beat complex adaptive retrieval systems on QA quality while being far cheaper and better calibrated for self-knowledge. The authors also analyze out-of-domain transfer of UE methods and measure functional complexity via Rademacher complexity and Hessian sharpness, and they release code and data.

Significance. If the central claims survive scrutiny, this would be a practically valuable benchmark: it quantifies the cost-performance trade-off of adaptive retrieval in terms of a directly actionable efficiency metric (LM calls), it evaluates self-knowledge rather than only downstream accuracy, and it covers a broad method space under a shared retriever and base model. The paper deserves credit for reporting bootstrap standard deviations, providing transparent efficiency counts, releasing code and data, and applying the same self-knowledge ground truth (no-context answer correctness) uniformly across methods, so the self-knowledge comparison is not circular with the methods being compared. The complexity analysis using Rademacher complexity and loss-landscape sharpness is an original addition. However, the headline claim of "comparable QA performance" on multi-hop datasets is currently overstated relative to the paper's own Table 1, and the UE numbers are partly an oracle selection over methods and classifiers, which limits what can be concluded about any single UE method.

major comments (4)
  1. The claim that UE methods "perform comparably on multi-hop datasets" is not supported on two of the three multi-hop datasets. In Table 1, Best UE achieves InAcc 0.384 on 2WikiMultiHopQA versus 0.454 for AdaptiveRAG and 0.456 for DRAGIN, a gap of 0.070-0.072; on MuSiQue, Best UE achieves 0.104 versus 0.140 and 0.134, a gap of 0.030-0.036. Appendix F reports bootstrap standard deviations of about 0.02 for 2Wiki and 0.01 for MuSiQue, so these gaps are roughly 2-3.5 standard errors, not noise. The appendix Table 15 shows an even larger MuSiQue gap against in-domain fine-tuned AdaptiveRAG (0.154). The abstract's "maintaining comparable QA performance" and Takeaway 1 should be revised to reflect that the comparable-performance claim holds only on HotpotQA, or the authors should provide a fixed UE configuration selected without test-set information that achieves these numbers.
  2. The "Best UE" row of Table 1 is a per-dataset oracle: it takes the maximum InAcc over 27 UE methods, and Appendix F states that the best classifier is selected based on downstream metrics. With 500 test questions per dataset, this procedure yields an upper envelope rather than the expected performance of any deployable single UE method. A fixed method such as EigValLaplacian with a fixed classifier would generally score lower, which could further widen the gaps on 2WikiMultiHopQA and MuSiQue. The authors should report results for a single a-priori UE method (or a nested selection rule) and treat "Best UE" explicitly as an upper bound, not as evidence that "uncertainty methods" as a class maintain comparable QA performance.
  3. The assertion that baseline methods "follow their original protocols" is contradicted by the reimplementation details in the appendix. Rowen is reimplemented with Qwen 2.5-72B-Instruct as the verification model instead of the original Qwen-Max-0428, and SeaKR is reimplemented with LLaMA 3.1-8B; Appendix G also states that SeaKR's eigenscore threshold is the one "because it gave the best results" after trying others. These deviations matter because a central conclusion is that simple UE methods outperform "complex pipelines": if the reimplementations underperform the original published systems, the comparison is not against the methods as originally proposed. The paper should either use original implementations, report the original authors' published numbers alongside the reimplementations, or explicitly discuss how the model and threshold changes affect the comparison.
  4. The paper primarily relies on In-Accuracy (whether the predicted answer contains the ground truth), but this metric is sensitive to verbosity, and the appendix data show a stark inconsistency for Rowen: on NQ, Rowen has InAcc 0.494 but EM 0.002 and F1 0.104; on TriviaQA, InAcc 0.656 with EM 0.022 and F1 0.188. The text in Section 4.3.1 says overall trends across metrics are "generally consistent", which is not true for these rows. Since Rowen's high InAcc is used indirectly to support the "comparable QA performance" narrative, the authors should either show that their conclusions hold under EM/F1 as well, or explain why a method with near-zero exact match should be considered to have comparable QA quality.
minor comments (5)
  1. Table 6 is captioned "Detailed QA performance results for uncertainty methods on one-hop datasets", but the table contains columns for 2WikiMultiHopQA, HotpotQA, and MuSiQue; the caption should say multi-hop datasets or split the table.
  2. Table 10 and Table 14 are identical descriptions of the uncertainty estimation methods; one of them should be removed to avoid duplication.
  3. The dataset name is misspelled as "Musicque" in Figures 8 and 16; it should be MuSiQue or Musique consistently with the rest of the paper.
  4. The sentence "EigValLaplacian also ranks highest for Retrieval Calls, indicating overconfidence" is ambiguous: in Figure 3, EigValLaplacian appears at the bottom of the RC rank list, i.e., it has the worst (largest) rank, not the best. Please rephrase to say it has the worst retriever-call rank.
  5. Appendix F says all experiments use LLaMA 3.1-8B with "default generation parameters", but consistency-based UE methods require sampling multiple responses; please report the sampling temperature, number of samples, and any decoding parameters used for the uncertainty scores, since these are not recoverable from "default".

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central comparison is an external empirical benchmark; the self-knowledge label and efficiency/QA metrics are defined independently of any method's output.

full rationale

The paper's derivation chain is an empirical benchmark rather than a formal derivation. The self-knowledge label is defined in Section 4.3.3 as the In-Accuracy of the model's response without external knowledge, and the same label is applied uniformly to every uncertainty method and every adaptive-retrieval baseline. Uncertainty scores are computed on training and test sets with the LM-Polygraph library, classifiers are trained on the training-set scores, and the resulting predictions are evaluated against the same external label. QA performance and efficiency are measured with standard metrics (In-Accuracy, EM, F1, retriever calls, LM calls) that do not depend on any method's internal construction. The 'Best UE' row is an optimistic per-dataset oracle over 27 uncertainty methods, but this is a reporting choice and a statistical fairness concern, not a case where a prediction is defined in terms of the conclusion. The use of LM-Polygraph is a self-citation by one co-author, but the uncertainty methods implemented there are established techniques from prior external work, the implementation is released code, and the paper's headline findings do not depend on an unverified theorem or ansatz imported from a self-citation. Potential weaknesses such as reimplemented baselines, the possible selection of classifiers based on downstream metrics, and the multi-hop performance gaps visible in Table 1 are correctness or evaluation-fidelity concerns, not circularity. I find no step in which an output is equivalent to an input by construction, and no load-bearing self-citation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The fitted choices in this benchmark are the per-dataset selection of the best UE method, the per-dataset selection of the best classifier among five, and the tuned SeaKR threshold. The central assumptions are the In-Accuracy-based definition of self-knowledge, the fidelity of baseline reimplementations, the adequacy of BM25, and the use of In-Accuracy as the primary QA metric. No new entities are postulated; the Hybrid method concatenates existing uncertainty features and has no independent falsifiable handle beyond the benchmark.

free parameters (3)
  • Best UE method selection = varies per dataset (e.g., EigValLaplacian for NQ)
    Table 1's 'Best UE' row picks the top-performing UE method per dataset after test evaluation; this is a post-hoc oracle selection, not a fixed deployable method, and inflates the comparison.
  • UE classifier selection = best of {Threshold, Logistic Regression, Decision Tree, KNN, MLP} per dataset
    Section 4.1 and Appendix F: 'the best classifier's performance reported based on downstream metrics'; choosing among 5 classifiers per method per dataset is a fitted model-selection choice.
  • SeaKR eigenscore threshold = same as original SeaKR paper, other values tried
    Appendix G: 'we use the same eigenscore threshold as in the original paper because it gave the best results, but we have also tried others'; threshold selection is a fitted choice.
assumptions (4)
  • domain assumption Self-knowledge ground truth equals In-Accuracy of the no-context response
    Section 4.3.3 defines the ground truth label y from In-Accuracy of the model's answer without external knowledge; if this operationalization is wrong (e.g., paraphrases count as wrong), all self-knowledge comparisons inherit the error.
  • ad hoc to paper Baseline methods' original hyperparameters and prompts remain valid when reimplemented with LLaMA 3.1-8B and a changed verification model
    Section 4.1 and Appendix F/G reimplement Rowen and SeaKR on LLaMA 3.1-8B and replace Qwen-Max-0428 with Qwen 2.5-72B-Instruct for Rowen; the paper assumes the original settings carry over.
  • domain assumption BM25 over the DPR Wikipedia corpus is an adequate retriever for all six QA datasets
    Section 4.1 uses BM25 with Elasticsearch and the Karpukhin et al. (2020) Wikipedia corpus for all methods and datasets, following prior work; a stronger or weaker retriever could change the relative ranking of retrieval-triggering methods.
  • domain assumption In-Accuracy is a sufficient primary QA metric and trends generalize to EM and F1
    Section 4.3.1 relies primarily on In-Accuracy, asserting 'the overall trends across these metrics are generally consistent'; Table 13 shows InAcc correlates only 0.63 with EM, so the chosen primary metric is a consequential modeling choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home." pith.science (2026). https://pith.science/paper/REP4XXFJ

@misc{pith2026250112835,
  author       = {Pith},
  title        = {Pith review of: Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/REP4XXFJ}},
  note         = {Machine review of arXiv:2501.12835}
}
read the original abstract

Retrieval Augmented Generation (RAG) improves correctness of Question Answering (QA) and addresses hallucinations in Large Language Models (LLMs), yet greatly increase computational costs. Besides, RAG is not always needed as may introduce irrelevant information. Recent adaptive retrieval methods integrate LLMs' intrinsic knowledge with external information appealing to LLM self-knowledge, but they often neglect efficiency evaluations and comparisons with uncertainty estimation techniques. We bridge this gap by conducting a comprehensive analysis of 35 adaptive retrieval methods, including 8 recent approaches and 27 uncertainty estimation techniques, across 6 datasets using 10 metrics for QA performance, self-knowledge, and efficiency. Our findings show that uncertainty estimation techniques often outperform complex pipelines in terms of efficiency and self-knowledge, while maintaining comparable QA performance.

Figures

Figures reproduced from arXiv: 2501.12835 by the authors.

Figure 1
Figure 1. Performance comparison of the state-of-the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Average overconfidence and underconfidence [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Uncertainty methods average ranks for In-Accuracy, ROC-AUC and Retrieval Calls. Smaller rank indicate [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The transferability of methods between datasets was evaluated using average changes in metrics for [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Average loss landscape sharpness in logarith [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Average QA performance drop for uncertainty methods for when switching maximum over classifiers to [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Performance comparison showing the relationship between LLM calls and Bootstrap Mean In-Accuracy. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Heatmap of improvement/decrease of the Accuracy and In-Accuracy scores on the OOD setup for the [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Heatmap of improvement/decrease of the Accuracy and In-Accuracy scores on the OOD setup for the [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Heatmap of improvement/decrease of the Accuracy and In-Accuracy scores on the OOD setup for the [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Heatmap of improvement/decrease of the In-Accuracy scores on the OOD setup for the SeaKR and [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Heatmap of improvement/decrease of the In-Accuracy scores on the OOD setup for the FLARE and [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Top-5 UE methods as a features for hybrid method across datasets. [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Feature Importance for each dataset for Hybrid method. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Feature Importance across datasets for Hybrid method. Different Uncertainty Estimation methods [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QA

    cs.CL 2025-05 conditional novelty 7.0 of 10

    EverGreenQA and EG-E5 provide a multilingual, human-labeled evergreen question classifier that improves self-knowledge estimation and QA dataset curation.

  2. HALT: Verification-Aware Stopping for Retrieval-Augmented Search Agents

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A per-hop evidence coverage verifier can stop multi-hop retrieval agents early, cutting search loops by up to 45% while preserving standardized-extractor exact match.

  3. Reconsidering LLM Uncertainty Estimation Methods in the Wild

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Most LLM uncertainty estimates degrade under distribution shift and adversarial prompts, but simple ensembling of scores at test time improves reliability.

  4. LLM-Independent Adaptive RAG: Let the Question Speak for Itself

    cs.CL 2025-05 conditional novelty 6.0 of 10

    External features such as entity popularity and question type can drive adaptive retrieval decisions without extra large-language-model calls, matching the accuracy of uncertainty-based methods on several QA benchmark...

  5. Fast, Slow, and Tool-augmented Thinking for LLMs: A Review

    cs.CL 2025-08 conditional novelty 5.0 of 10

    LLM reasoning strategies are organized along fast/slow and internal/external boundaries, and recent adaptive selection methods are surveyed.

Reference graph

Works this paper leans on

56 extracted references · 9 canonical work pages · cited by 5 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Zeyuan Allen-Zhu and Yuanzhi Li. 2024. https://arxiv.org/abs/2309.14316 Physics of language models: Part 3.1, knowledge storage and extraction . Preprint, arXiv:2309.14316

    arXiv 2024

  4. [4]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. https://openreview.net/forum?id=hSyW5go0v8 Self-rag: Learning to retrieve, generate, and critique through self-reflection . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  5. [5]

    Park, and Sung Ju Hwang

    Jinheon Baek, Soyeong Jeong, Minki Kang, Jong C. Park, and Sung Ju Hwang. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.107 Knowledge-augmented language model verification . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pages 1720--1736. Association for Computati...

    doi:10.18653/v1/2023.emnlp-main.107 2023

  6. [6]

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, ...

    2022

  7. [7]

    Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Ga \" e l Varoquaux. 2013. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Wor...

    2013

  8. [8]

    Maxime Darrin, Pablo Piantanida, and Pierre Colombo. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.357 Rainproof: An umbrella to shield text generator from out-of-distribution data . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pages 5831--5857. Association for ...

    doi:10.18653/v1/2023.emnlp-main.357 2023

Show all 56 references
  1. [9]

    Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng. 2024. https://doi.org/10.48550/ARXIV.2402.10612 Retrieve only when it needs: Adaptive retrieval augmentation for hallucination mitigation in large language models . CoRR, abs/2402.10612

    2024 doi

  2. [10]

    Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2023. Shifting attention to relevance: Towards the uncertainty estimation of large language models. arXiv preprint arXiv:2307.01379

    2023 arXiv

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  4. [12]

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. 2024 a . https://doi.org/10.18653/V1/2024.FINDINGS-ACL.558 Fact-checking...

    2024 doi

  5. [13]

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, et al. 2024 b . Fact-checking the output of large language models via token-level uncertainty quantifica...

    2024 arXiv

  6. [14]

    Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. https://doi.org/10.18653/V1/2023.EMNLP-DEMO.41 Lm-polygraph:...

    2023 doi

  7. [15]

    Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Fr \'e d \'e ric Blain, Francisco Guzm \'a n, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computation...

    2020

  8. [16]

    Xavier Glorot and Yoshua Bengio. 2010. https://proceedings.mlr.press/v9/glorot10a.html Understanding the difficulty of training deep feedforward neural networks . In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of P...

    2010

  9. [17]

    Xanh Ho, Anh - Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/V1/2020.COLING-MAIN.580 Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational ...

    2020 doi

  10. [18]

    Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. 2024. https://doi.org/10.18653/V1/2024.NAACL-LONG.389 Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity . In Proceedings of the 2024 Conference of the Nor...

    2024 doi

  11. [19]

    Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang, Lifeng Shang, Ruiming Tang, Qun Liu, and Wei Wang. 2024. https://arxiv.org/abs/2402.11905 Learning to edit: Aligning llms with knowledge editing . Preprint, arXiv:2402.11905

    2024 arXiv

  12. [20]

    Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.495 Active retrieval augmented generation . In Proceedings of the 2023 Conference on Empirical Methods in...

    2023 doi

  13. [21]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/V1/P17-1147 Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  14. [22]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  15. [23]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen - tau Yih. 2020. https://doi.org/10.18653/V1/2020.EMNLP-MAIN.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on...

    2020 doi

  16. [24]

    I Can't Believe It's Not Better! - Understanding Deep Learning Through Empirical Falsification

    Simran Kaur, Jeremy Cohen, and Zachary Chase Lipton. 2023. https://proceedings.mlr.press/v187/kaur23a.html On the maximum hessian eigenvalue and generalization . In Proceedings on "I Can't Believe It's Not Better! - Understanding Deep Learning Through Empirical Falsification" ...

    2023

  17. [25]

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. https://openreview.net/forum?id=HklBjCEKvH Generalization through memorization: Nearest neighbor language models . In International Conference on Learning Representations

    2020

  18. [26]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. https://openreview.net/forum?id=VD-AYtP0dve Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation . In The Eleventh International Conference on Learning Representations, ICLR...

    2023

  19. [28]

    Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. https://proceedings.neurips.cc/paper/2018/hash/abdeb6f575ac5c6676b747bca8d09cc2-Abstract.html A simple unified framework for detecting out-of-distribution samples and adversarial attacks . In Advances in Neural Informat...

    2018

  20. [29]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020 a . https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205...

    2020

  21. [30]

    u ttler, Mike Lewis, Wen - tau Yih, Tim Rockt \

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \" u ttler, Mike Lewis, Wen - tau Yih, Tim Rockt \" a schel, Sebastian Riedel, and Douwe Kiela. 2020 b . https://proceedings.neurips.cc/paper/2020/hash/6b493230205f7...

    2020

  22. [31]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187

    2023 arXiv

  23. [32]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/V1/2023.ACL-LONG.546 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . In Proceedings of the 6...

    2023 doi

  24. [33]

    Shiyu Ni, Keping Bi, Jiafeng Guo, and Xueqi Cheng. 2024. When do llms need retrieval augmentation? mitigating llms' overconfidence helps retrieval augmentation. arXiv preprint arXiv:2402.11457

    2024 arXiv

  25. [34]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

    2020

  26. [35]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/V1/D16-1264 Squad: 100, 000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, A...

    2016 doi

  27. [36]

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. https://doi.org/10.1162/tacl_a_00605 In-context retrieval-augmented language models . Transactions of the Association for Computational Linguistics, 11:1316--1331

    2023 doi

  28. [37]

    Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J. Liu. 2023. https://openreview.net/forum?id=kJUS5nD0vPB Out-of-distribution detection and selective generation for conditional language models . In The Eleventh International C...

    2023

  29. [38]

    Robertson, Steve Walker, Susan Jones, Micheline Hancock - Beaulieu, and Mike Gatford

    Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock - Beaulieu, and Mike Gatford. 1994. http://trec.nist.gov/pubs/trec3/papers/city.ps.gz Okapi at TREC-3 . In Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-...

    1994

  30. [39]

    Levent Sagun, Leon Bottou, and Yann LeCun. 2016. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476

    2016 arXiv

  31. [40]

    Chi, Nathanael Sch \" a rli, and Denny Zhou

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Sch \" a rli, and Denny Zhou. 2023. https://proceedings.mlr.press/v202/shi23a.html Large language models can be easily distracted by irrelevant context . In International Conference on Mac...

    2023

  32. [41]

    Adi Simhi, Jonathan Herzig, Idan Szpektor, and Yonatan Belinkov. 2024. https://arxiv.org/abs/2404.09971 Constructing benchmarks and interventions for combating hallucinations in llms . Preprint, arXiv:2404.09971

    2024 arXiv

  33. [42]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170 Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on Emp...

    2013

  34. [43]

    Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024 a . https://arxiv.org/abs/2403.10081 Dragin: Dynamic retrieval augmented generation based on the information needs of large language models . Preprint, arXiv:2403.10081

    2024 arXiv

  35. [44]

    Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024 b . https://doi.org/10.18653/V1/2024.ACL-LONG.702 DRAGIN: dynamic retrieval augmented generation based on the real-time information needs of large language models . In Proceedings of the 62nd Annual Meeting o...

    2024 doi

  36. [45]

    Junya Takayama and Yuki Arase. 2019. Relevant and informative response generation using pointwise mutual information. In Proceedings of the First Workshop on NLP for Conversational AI, pages 133--138

    2019

  37. [46]

    Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, and Bolin Ding. 2024. When to trust llms: Aligning confidence with response quality. arXiv preprint arXiv:2404.17287

    2024 arXiv

  38. [47]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  39. [48]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://doi.org/10.1162/TACL\_A\_00475 Musique: Multihop questions via single-hop question composition . Trans. Assoc. Comput. Linguistics, 10:539--554

    2022 doi

  40. [49]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. https://doi.org/10.18653/V1/2023.ACL-LONG.557 Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions . In Proceedings of the 61st Annual Meeting of th...

    2023 doi

  41. [50]

    Liam van der Poel, Ryan Cotterell, and Clara Meister. 2022. https://doi.org/10.18653/V1/2022.EMNLP-MAIN.399 Mutual information alleviates hallucinations in abstractive summarization . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EM...

    2022 doi

  42. [51]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  43. [52]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/V1/D18-1259 Hotpotqa: A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  44. [53]

    Zijun Yao, Weijian Qi, Liangming Pan, Shulin Cao, Linmei Hu, Weichuan Liu, Lei Hou, and Juanzi Li. 2024. https://doi.org/10.48550/ARXIV.2406.19215 Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation . CoRR, abs/2406.19215

    2024 doi arXiv

  45. [54]

    Dong Yin, Ramchandran Kannan, and Peter Bartlett. 2019. Rademacher complexity for adversarially robust generalization. In International conference on machine learning, pages 7085--7094. PMLR

    2019

  46. [55]

    Xunjian Yin, Xu Zhang, Jie Ruan, and Xiaojun Wan. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.124 Benchmarking knowledge boundary for large language models: A different perspective on model evaluation . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

    2024 doi

  47. [56]

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don't know? arXiv preprint arXiv:2305.18153

    2023 arXiv

  48. [57]

    KiYoon Yoo, Jangho Kim, Jiho Jang, and Nojun Kwak. 2022. https://doi.org/10.18653/V1/2022.FINDINGS-ACL.289 Detection of adversarial examples in text classification: Benchmark and baseline via robust density estimation . In Findings of the Association for Computational Linguist...

    2022 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.