{"id":"a5e1e529-ed7a-4753-8b7e-6f2c609e6bb9","arxiv_id":"2501.12835","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A benchmark of 35 retrieval-decision methods on 6 QA datasets finds uncertainty estimation rivals or beats adaptive RAG pipelines in accuracy, with far lower compute and stronger self-knowledge scores.","lead":"This paper compares 35 ways of deciding when a question-answering system should look up external information before responding. It finds that simple, established uncertainty measures are often cheaper and better at telling when the model lacks knowledge, with answer accuracy comparable to recent complex retrieval pipelines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main Table 1 contradicts 'comparable QA performance' on 2Wiki and MuSiQue: the best UE underperforms AdaptiveRAG/DRAGIN by 0.07 and 0.036 InAcc, so the central claim is overstated.","rationale":"The reader's weakest assumption focused on baseline reimplementation fidelity (Rowen/SeaKR using different verification models). That is a legitimate concern, but it is not the most load-bearing one: the multi-hop QA gap that contradicts 'comparable performance' comes from AdaptiveRAG and DRAGIN, whose implementations do not depend on the changed Qwen verification model. The reader's rationale did note that Table 15's in-domain results exceed the best UE on 2Wiki and MuSiQue, but the same contradiction is already present in the main Table 1 with non-fine-tuned baselines, which the reader did not emphasize. I therefore partially agree with the reader's weakest assumption but identify a sharper internal inconsistency. The paper is still a valuable benchmark: the efficiency advantage (2 vs 5–80 LM calls) and self-knowledge advantage of UE methods are well supported, and the 'Ideal' oracle shows large headroom for all approaches. The required fix is a precise, evidence-calibrated restatement of the QA-comparability claim, not a rejection of the benchmark or its efficiency findings. Hence the existing CONDITIONAL verdict remains appropriate.","tokens_in":34659,"tokens_out":5981,"duration_ms":60855,"concrete_test":"Re-analyze Table 1 using the released code: for 2WikiMultiHopQA and MuSiQue, compute paired bootstrap 95% confidence intervals for the InAcc difference between the best UE method and both AdaptiveRAG and DRAGIN, and run McNemar tests on the binary InAcc labels. If the confidence interval excludes zero on both datasets, revise the abstract and Takeaway 1 to state that UE methods are comparable on single-hop and HotpotQA but substantially worse on 2Wiki and MuSiQue. If the interval includes zero, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 5.1 claim that uncertainty estimation (UE) methods maintain 'comparable QA performance' with complex adaptive-retrieval pipelines on multi-hop datasets. This is contradicted by the paper's own Table 1. On 2WikiMultiHopQA, the best reported UE method (EigValLaplacian, InAcc 0.384) trails AdaptiveRAG (0.454) and DRAGIN (0.456) by a margin of 0.070–0.072. On MuSiQue, the best UE (0.104) is 0.036 below AdaptiveRAG (0.140) and 0.030 below DRAGIN (0.134). The reported bootstrap standard deviations are about 0.02, so these gaps are roughly 1.5–3.5 standard errors, not noise. The gap is not driven by reimplemented baselines such as Rowen or SeaKR: the methods that beat UE on these datasets are AdaptiveRAG and DRAGIN, whose protocols do not depend on the changed verification model. The appendix Table 15 shows an even larger MuSiQue gap (AdaptiveRAG 0.154) for in-domain fine-tuning, but the problem is already visible in the main table. Moreover, 'Best UE' is a per-dataset oracle over 27 uncertainty methods, so the reported 0.384 is an upper bound on what a single UE method would achieve; a fixed UE method would likely be lower. Thus the claim of comparable multi-hop QA performance is unsupported on two of the three multi-hop datasets, and the paper's headline recommendation to system builders—similar accuracy at much lower cost—does not hold for those datasets.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large-scale empirical comparison of 8 adaptive retrieval pipelines and 27 uncertainty estimation (UE) methods on 6 QA datasets, evaluating QA accuracy (In-Accuracy, EM, F1), self-knowledge identification (ROC-AUC, Spearman correlation, accuracy, over/underconfidence), and efficiency (number of LM calls and retriever calls). The central claim is that simple UE methods often match or beat complex adaptive retrieval systems on QA quality while being far cheaper and better calibrated for self-knowledge. The authors also analyze out-of-domain transfer of UE methods and measure functional complexity via Rademacher complexity and Hessian sharpness, and they release code and data.","tokens_in":34819,"tokens_out":6624,"duration_ms":70045,"significance":"If the central claims survive scrutiny, this would be a practically valuable benchmark: it quantifies the cost-performance trade-off of adaptive retrieval in terms of a directly actionable efficiency metric (LM calls), it evaluates self-knowledge rather than only downstream accuracy, and it covers a broad method space under a shared retriever and base model. The paper deserves credit for reporting bootstrap standard deviations, providing transparent efficiency counts, releasing code and data, and applying the same self-knowledge ground truth (no-context answer correctness) uniformly across methods, so the self-knowledge comparison is not circular with the methods being compared. The complexity analysis using Rademacher complexity and loss-landscape sharpness is an original addition. However, the headline claim of \"comparable QA performance\" on multi-hop datasets is currently overstated relative to the paper's own Table 1, and the UE numbers are partly an oracle selection over methods and classifiers, which limits what can be concluded about any single UE method.","major_comments":[{"comment":"The claim that UE methods \"perform comparably on multi-hop datasets\" is not supported on two of the three multi-hop datasets. In Table 1, Best UE achieves InAcc 0.384 on 2WikiMultiHopQA versus 0.454 for AdaptiveRAG and 0.456 for DRAGIN, a gap of 0.070-0.072; on MuSiQue, Best UE achieves 0.104 versus 0.140 and 0.134, a gap of 0.030-0.036. Appendix F reports bootstrap standard deviations of about 0.02 for 2Wiki and 0.01 for MuSiQue, so these gaps are roughly 2-3.5 standard errors, not noise. The appendix Table 15 shows an even larger MuSiQue gap against in-domain fine-tuned AdaptiveRAG (0.154). The abstract's \"maintaining comparable QA performance\" and Takeaway 1 should be revised to reflect that the comparable-performance claim holds only on HotpotQA, or the authors should provide a fixed UE configuration selected without test-set information that achieves these numbers.","section":null},{"comment":"The \"Best UE\" row of Table 1 is a per-dataset oracle: it takes the maximum InAcc over 27 UE methods, and Appendix F states that the best classifier is selected based on downstream metrics. With 500 test questions per dataset, this procedure yields an upper envelope rather than the expected performance of any deployable single UE method. A fixed method such as EigValLaplacian with a fixed classifier would generally score lower, which could further widen the gaps on 2WikiMultiHopQA and MuSiQue. The authors should report results for a single a-priori UE method (or a nested selection rule) and treat \"Best UE\" explicitly as an upper bound, not as evidence that \"uncertainty methods\" as a class maintain comparable QA performance.","section":null},{"comment":"The assertion that baseline methods \"follow their original protocols\" is contradicted by the reimplementation details in the appendix. Rowen is reimplemented with Qwen 2.5-72B-Instruct as the verification model instead of the original Qwen-Max-0428, and SeaKR is reimplemented with LLaMA 3.1-8B; Appendix G also states that SeaKR's eigenscore threshold is the one \"because it gave the best results\" after trying others. These deviations matter because a central conclusion is that simple UE methods outperform \"complex pipelines\": if the reimplementations underperform the original published systems, the comparison is not against the methods as originally proposed. The paper should either use original implementations, report the original authors' published numbers alongside the reimplementations, or explicitly discuss how the model and threshold changes affect the comparison.","section":null},{"comment":"The paper primarily relies on In-Accuracy (whether the predicted answer contains the ground truth), but this metric is sensitive to verbosity, and the appendix data show a stark inconsistency for Rowen: on NQ, Rowen has InAcc 0.494 but EM 0.002 and F1 0.104; on TriviaQA, InAcc 0.656 with EM 0.022 and F1 0.188. The text in Section 4.3.1 says overall trends across metrics are \"generally consistent\", which is not true for these rows. Since Rowen's high InAcc is used indirectly to support the \"comparable QA performance\" narrative, the authors should either show that their conclusions hold under EM/F1 as well, or explain why a method with near-zero exact match should be considered to have comparable QA quality.","section":null}],"minor_comments":[{"comment":"Table 6 is captioned \"Detailed QA performance results for uncertainty methods on one-hop datasets\", but the table contains columns for 2WikiMultiHopQA, HotpotQA, and MuSiQue; the caption should say multi-hop datasets or split the table.","section":null},{"comment":"Table 10 and Table 14 are identical descriptions of the uncertainty estimation methods; one of them should be removed to avoid duplication.","section":null},{"comment":"The dataset name is misspelled as \"Musicque\" in Figures 8 and 16; it should be MuSiQue or Musique consistently with the rest of the paper.","section":null},{"comment":"The sentence \"EigValLaplacian also ranks highest for Retrieval Calls, indicating overconfidence\" is ambiguous: in Figure 3, EigValLaplacian appears at the bottom of the RC rank list, i.e., it has the worst (largest) rank, not the best. Please rephrase to say it has the worst retriever-call rank.","section":null},{"comment":"Appendix F says all experiments use LLaMA 3.1-8B with \"default generation parameters\", but consistency-based UE methods require sampling multiple responses; please report the sampling temperature, number of samples, and any decoding parameters used for the uncertainty scores, since these are not recoverable from \"default\".","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful benchmark, but the main quantitative story currently rests on two optimistic choices: the per-dataset \"Best UE\" oracle and the best-classifier selection, and on reimplementations of Rowen/SeaKR that deviate from the originals. The authors can address this by reporting a fixed UE configuration, tempering the multi-hop comparability claim, and providing a fidelity check against original published numbers. If those changes are made, the paper would be a solid contribution to the adaptive-retrieval evaluation literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a solid benchmark with an overstated headline. It compares 27 uncertainty estimation methods against 8 adaptive retrieval pipelines on 6 QA datasets under one protocol (LLaMA 3.1-8B, BM25), reports bootstrap errors, and releases code and data. That alone makes it worth reading.\n\nThe claim that UE methods maintain 'comparable QA performance' on multi-hop QA is the load-bearing finding, and the paper's own Table 1 undercuts it. On 2Wiki, the best UE (EigValLaplacian, 0.384 InAcc) is 0.07 behind AdaptiveRAG and DRAGIN; with bootstrap SDs near 0.02, that's roughly 3.5 standard errors. On MuSiQue the best UE is 0.104 vs 0.140 for AdaptiveRAG and 0.134 for DRAGIN. Only on HotpotQA are they actually comparable. So the abstract and Takeaway 1 should be reworded to 'comparably on some multi-hop datasets, clearly worse on others.'\n\nThe 'Best UE' row is an oracle over 27 methods, so those numbers are upper bounds. A single fixed UE method would likely be lower. Also, the classifier per UE method is chosen as the best of five based on downstream metrics, which is another test-set selection. That inflates the UE side somewhat.\n\nThe reimplemented baselines (Rowen with Qwen 2.5-72B instead of Qwen-Max, SeaKR on LLaMA 3.1-8B) are a legitimate weakness, but they are not what drives the multi-hop gap: AdaptiveRAG and DRAGIN follow their original protocols. Table 15 in the appendix, with in-domain fine-tuned AdaptiveRAG, shows an even larger MuSiQue gap (0.154 vs 0.104), so the concern is robust.\n\nWhat the paper does well: the efficiency accounting (LM and retriever calls) is transparent and favors UE methods; the self-knowledge metrics (ROC-AUC, correlation, over/underconfidence) are a genuine addition; OOD transfer across dataset pairs and Rademacher complexity analyses are real, if exploratory. The citation practice is fine; the UE methods are established, and the baselines cite their sources.\n\nMy recommendation: send it to review. The benchmark is reusable, the central question is important, and the flaws are correctable in revision. The authors need to either soften the 'comparable' claim or stratify it by dataset, report a fixed UE method rather than just the oracle, and address the Table 15 discrepancy. I would not cite this as-is for the multi-hop claim, but I would cite it for the benchmark and the self-knowledge comparison.","headline":"A useful, well-run benchmark whose headline 'comparable QA performance' is contradicted by the paper's own Table 1 on two of three multi-hop datasets.","tokens_in":35559,"tokens_out":3368,"would_cite":true,"duration_ms":29635,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard uncertainty scores often beat bespoke adaptive-retrieval pipelines on efficiency and self-knowledge while matching their answer accuracy.","keywords":["adaptive retrieval","uncertainty estimation","self-knowledge","retrieval-augmented generation","question answering","efficiency","out-of-distribution transfer"],"falsifier":"Run the same 35-method comparison with the original released implementations and verification models for the baselines (notably Rowen and SeaKR), or repeat on a second base LLM; if the uncertainty methods cease to match the pipelines' QA accuracy while using fewer calls, the central claim is false.","tokens_in":34262,"feed_emoji":"🧠","tokens_out":13162,"duration_ms":106930,"temperature":0.7,"pith_summary":"Adaptive retrieval systems try to decide, question by question, whether a language model should consult external documents or answer from memory. The paper asks whether the elaborate pipelines built for this decision are actually better than plain uncertainty scores, and finds that they usually are not: across six QA datasets, 27 standard uncertainty-estimation techniques matched or outperformed eight recent adaptive-retrieval methods on answer accuracy while using far fewer language-model and retriever calls. The uncertainty methods also did better at self-knowledge, the ability to tell apart questions the model can answer from those it cannot. The practical upshot is that a simple confidence score plus a threshold may be the most efficient known way to gate retrieval, and that better self-knowledge, not better retrieval, is what still separates current systems from an oracle that retrieves only when necessary.","feed_headline":"Simple confidence scores often beat elaborate retrieval pipelines","feed_subtitle":"On six QA datasets, they match answer accuracy while using far fewer calls and better judging self-knowledge.","key_machinery":"The load-bearing mechanism is the self-knowledge score: each method is viewed as a function $f$ mapping an input question to a real number, where higher values indicate lower self-knowledge, and a classifier $C$ decides whether to retrieve. For uncertainty estimation methods this score comes from a standard uncertainty measure (mean or max token entropy, lexical similarity of sampled answers, Laplacian eigenvalue sums, or token-relevance reweighted entropy), and the classifier is trained once on a small set of questions and then applied at test time. Efficiency is counted as the number of language-model calls and retriever calls per question, and the paper contrasts every method against two anchors: 'Best UE', the top uncertainty score per dataset, and 'Ideal', an oracle that knows exactly when retrieval is needed, which bounds how much room remains for better self-knowledge.","core_discovery":"The paper's central discovery is that established uncertainty estimation (UE) techniques---simple scores computed from token probabilities, sampled-response consistency, or internal states---are often more useful for adaptive retrieval than the elaborate pipelines built for the task. Across 35 methods and 6 QA datasets, UE methods outperform the eight recent adaptive-retrieval baselines on single-hop questions and match them on multi-hop questions, while using two or fewer language-model calls and at most one retriever call per question, compared with pipelines that routinely make several to dozens of calls. UE methods also rank first or second on self-knowledge metrics, meaning their scores better separate questions the model can answer from those it cannot. No single method dominates: internal-state based SeaKR is strongest on single-hop self-knowledge, while the trainable reflexive classifier in AdaptiveRAG handles multi-hop questions better, and the paper reports that downstream accuracy and self-knowledge are only weakly correlated, so QA performance alone does not reveal whether a system knows when it needs external help.","pith_inferences":["A pragmatic takeaway the authors leave implicit: a production system could replace an adaptive-retrieval module with a single uncertainty score (e.g., mean entropy or lexical similarity) plus a threshold, and the paper's numbers suggest this would capture most of the accuracy benefit of the pipelines at a fraction of the compute.","The weak correlation between QA performance and self-knowledge implies that benchmark rankings of adaptive-retrieval methods are incomplete unless they also report calibration; a method could look strong on accuracy while systematically retrieving answers the model already knows.","Because internal-state methods showed the highest functional complexity and the largest out-of-domain self-knowledge drops, the simpler logit-based scores may generalize better as defaults; this is an inference from the paper's complexity and transfer analyses, not one of its stated conclusions.","A natural next experiment would be to combine a cheap uncertainty gate with a single retrieval pass and compare to the 'Ideal' bound; the paper's tables show an In-Accuracy gap between the best real method and the oracle on every dataset, suggesting headroom for better self-knowledge estimators."],"forward_implications":["A QA system can be made adaptive by thresholding a standard uncertainty score instead of running a multi-stage retrieval pipeline; on the six datasets tested this preserves answer accuracy while cutting language-model calls to two or fewer and retriever calls to at most one per question.","On single-hop questions the uncertainty-based gate is the best-performing adaptive strategy, and on multi-hop questions it matches the pipelines, so the complexity of bespoke adaptive retrieval buys little measured accuracy in this setup.","Because QA accuracy and self-knowledge are only weakly correlated (a moderate Spearman correlation of 0.65 between In-Accuracy and ROC-AUC ranks), methods should be evaluated on both axes; a system can score well on answers while routinely misjudging when it needs retrieval.","Out-of-domain transfer of uncertainty methods is stable for answer accuracy (typical loss under 4 percent), but self-knowledge accuracy degrades more and retriever calls increase, so deployment on a new distribution should budget for more retrieval.","The 'Ideal' oracle outperforms every real method on both accuracy and efficiency, indicating that better self-knowledge estimation, not better retrieval, is the remaining bottleneck for adaptive QA."],"supporting_citations":[{"why":"It supplies the lexical-similarity and mean/max entropy uncertainty scores that serve as the paper's strongest uncertainty baselines.","marker":"Fomicheva et al., 2020"},{"why":"It supplies EigValLaplacian and other consistency-based uncertainty scores used for adaptive retrieval.","marker":"Lin et al., 2023"},{"why":"It supplies SAR, the token-relevance reweighted entropy score used in the comparison.","marker":"Duan et al., 2023"},{"why":"It provides the AdaptiveRAG baseline and the protocol and prompt used to fit classifiers for the uncertainty methods.","marker":"Jeong et al., 2024"},{"why":"It provides the FLARE baseline, the token-probability-triggered retrieval method compared against.","marker":"Jiang et al., 2023"},{"why":"It provides the DRAGIN baseline with attention-based query reformulation.","marker":"Su et al., 2024b"},{"why":"It provides the Rowen baseline, the consistency-based adaptive retrieval method reimplemented with a changed verification model.","marker":"Ding et al., 2024"},{"why":"It provides the SeaKR baseline, the internal-state uncertainty module and reranker compared in the study.","marker":"Yao et al., 2024"},{"why":"It provides the IRCoT baseline and the interleaved chain-of-thought retrieval protocol used for multi-hop questions.","marker":"Trivedi et al., 2023"},{"why":"It provides the uncertainty-estimation implementation used to compute all 27 UE scores.","marker":"Fadeeva et al., 2023"}],"fun_headline_variants":["Simple confidence beats complex adaptive retrieval pipelines","Cheap uncertainty scores outperform fancy RAG designs","Adaptive retrieval: simple wins over elaborate systems","Efficiency and self-knowledge: uncertainty trumps complexity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the reimplemented versions of the eight baseline pipelines, all run on LLaMA 3.1-8B and with Rowen's consistency-checking model swapped from Qwen-Max-0428 to Qwen 2.5-72B-Instruct, faithfully represent the original methods.","fun_headline_variants_meta":{"raw":{"variants":["Simple confidence beats complex adaptive retrieval pipelines","Cheap uncertainty scores outperform fancy RAG designs","Adaptive retrieval: simple wins over elaborate systems","Efficiency and self-knowledge: uncertainty trumps complexity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000826,"raw_usage":{"total_tokens":3573,"prompt_tokens":871,"completion_tokens":2702,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":2643}},"tokens_in":487,"tokens_out":2702,"duration_ms":24052,"temperature":1.0,"reasoning_tokens":2643,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:44:25.792625+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 35-method comparison with the original released implementations and verification models for the baselines (notably Rowen and SeaKR), or repeat on a second base LLM; if the uncertainty methods cease to match the pipelines' QA accuracy while using fewer calls, the central claim is false.","supporting_citations":[],"review_version":1}