{"id":"06148213-b3d1-4e4e-8396-ed25f09bc249","arxiv_id":"2501.04899","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"SUGAR uses semantic entropy thresholds to dynamically switch between no retrieval, single-step retrieval, and multi-step retrieval, and reports accuracy gains on QA benchmarks.","lead":"This paper introduces SUGAR, a retrieval strategy that uses semantic entropy to decide when a question-answering model should fetch external documents, and whether one or multiple retrieval rounds are needed. The paper reports accuracy gains on five QA benchmarks, but the evaluation uses different base models for SUGAR and its baselines, and the decision thresholds are tuned on the test data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main comparison is confounded: SUGAR uses Llama-2-chat while single-step/no-retrieval baselines use FLAN-T5-XL, so reported gains may reflect generator strength rather than the retrieval policy.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the main comparison does not hold the generator fixed. SUGAR runs on Llama-2-chat (7B) while the Single-step and No Retrieval baselines run on FLAN-T5-XL, so the core accuracy comparison in Tables I and II is confounded. This is not a minor implementation detail; it is the central evidence for the paper's claim that semantic entropy improves QA performance. The within-model ablation partially addresses this for a restricted setting, but it does not rescue the headline comparison against all adaptive baselines. I also note a secondary issue: thresholds are selected via cross-validation on the evaluation datasets (§III-C), which risks fitting to the test data and would need to be addressed in any revision. However, the generator mismatch is the single most decisive flaw because it is unambiguous and directly invalidates the main tables. A controlled re-run with a shared generator and held-out threshold selection would settle whether the central claim has any empirical basis. Until then, the reader's REJECT verdict remains appropriate.","tokens_in":7374,"tokens_out":6301,"duration_ms":63294,"concrete_test":"Rebuild Tables I and II with all methods using the same generator (Llama-2-chat 7B) and the same retriever (Contriever-MS MARCO), with identical one-shot prompting and evaluation metrics, and select SUGAR's thresholds on a held-out validation split before any test-set evaluation. If SUGAR does not consistently beat a same-generator single-step baseline on SQuAD, NQ, TriviaQA, HotpotQA, and 2WikiMultiHopQA, the reported advantage cannot be attributed to the semantic-entropy retrieval strategy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that SUGAR outperforms naive single-step retrieval and other adaptive RAG approaches—is not supported by the main tables because the comparison is not controlled for the generator. Section III-B states that the No Retrieval and Single-step baselines use off-the-shelf FLAN-T5-XL, while Section III-C states that SUGAR uses Llama-2-chat (7B) as the generator. Thus Tables I and II vary both the retrieval policy and the underlying language model simultaneously. Since open-domain QA accuracy is highly sensitive to the base model's parametric knowledge and instruction tuning, the observed improvements could simply be a Llama-2-chat advantage over FLAN-T5-XL rather than a benefit of semantic-entropy-guided retrieval. The within-model ablation in Table III is a useful partial control, but it covers only TriviaQA and SQuAD, uses thresholds that appear to be tuned on those datasets, and does not compare against Adaptive Retrieval, Self-RAG, or Adaptive-RAG. A reader cannot currently attribute the headline results to SUGAR's retrieval decision rule.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SUGAR, an adaptive retrieval-augmented generation strategy for open-domain QA that uses semantic entropy to decide whether to retrieve, and if so, whether to perform single-step or multi-step retrieval. The method is training-free and is evaluated on five QA benchmarks against no-retrieval, single-step retrieval, and several adaptive RAG baselines. The authors report accuracy gains on some datasets and a reduction in retrieval steps, while acknowledging an inference-time trade-off. The central empirical claim is that semantic-entropy-guided retrieval improves QA accuracy and efficiency relative to naive and adaptive retrieval baselines.","tokens_in":7642,"tokens_out":5607,"duration_ms":58416,"significance":"The idea of replacing predictive entropy with semantic entropy as the trigger for retrieval is well motivated and, to my knowledge, not previously applied in exactly this way. The within-model ablation in Table III provides suggestive evidence that semantic entropy is a better confidence signal than predictive entropy for the same generator. If the headline comparisons were properly controlled, the method would be a useful and simple addition to adaptive RAG. However, as presented, the main tables do not support the stated claims because the generator is not held constant and the thresholds are tuned on the same benchmarks used for evaluation.","major_comments":[{"comment":"The main comparison is confounded: Section III-B states that the No Retrieval and Single-step baselines use FLAN-T5-XL, while Section III-C states that SUGAR uses Llama-2-chat (7B) as the generator. Tables I and II therefore vary both the retrieval policy and the base model, so the observed improvements over No Retrieval and Single-step retrieval cannot be attributed to SUGAR's decision rule. The authors should rerun all baselines with Llama-2-chat (7B) or run SUGAR with FLAN-T5-XL, and ideally also report the adaptive baselines on the same generator.","section":"III-B and III-C"},{"comment":"The semantic entropy thresholds are determined by cross-validation on the same datasets that are later used for reporting the main results in Tables I and II. This makes the headline numbers fitted values, not predictions, and contradicts the claim in Section I that the approach is \"not task- or data-dependent.\" The paper should report thresholds selected on a held-out validation set or on separate datasets, and show sensitivity to the threshold choice.","section":"III-C"},{"comment":"The claim that SUGAR outperforms naive single-step retrieval is not supported on 2WikiMultiHopQA: SUGAR achieves 39.75 accuracy while Single-step retrieval achieves 42.80, and IRCoT achieves 55.40. On HotpotQA, SUGAR's EM and F1 (38.77 and 49.85) are below those of the Multi-step IRCoT baseline (44.60 and 56.54), so the accuracy-only framing overstates the comparison. The multi-hop results need to be reported and discussed with the same metrics as the baselines.","section":"Table II"},{"comment":"The efficiency claim is not supported by the reported wall-clock times. On SQuAD, SUGAR takes 4.43 relative time versus 1.00 for single-step retrieval, and on TriviaQA it takes 3.13 versus 1.00, despite using fewer retrieval steps on TriviaQA. The paper's own text admits that \"inference for SUGAR takes longer than other adaptive methods for single-hop datasets.\" The efficiency advantage should be stated only in terms of retrieval steps, or the time measurements should include the cost of semantic entropy estimation and be compared on a common basis.","section":"III-C"},{"comment":"The ablation study is a useful partial control, but it does not repair the main comparison: it covers only TriviaQA and SQuAD, uses thresholds tuned on those datasets, and does not compare SUGAR against Adaptive Retrieval, Self-RAG, or Adaptive-RAG with the same generator. The claim that SUGAR \"helps mitigate overconfidence\" is not directly tested; the table reports accuracy and steps, not calibration or confidence. The ablation should be extended to all datasets and to the adaptive baselines, and should report significance or variance.","section":"III-D and Table III"}],"minor_comments":[{"comment":"The row labels in Table I are confusing: \"Single-hop Simple No Retrieval\" and the \"Data Types\" column are not explained, and the multi-hop Table II has rows that run together visually.","section":"Table I"},{"comment":"Equation (1) uses the notation |C| and C_i without defining whether C is the set of clusters, the number of clusters, or a specific cluster; this should be clarified.","section":"II-B, Eq. (1)"},{"comment":"References [18]--[22] concern robotics, EEG classification, and computer vision, and appear unrelated to the discussion of LLMs being distracted by noisy retrieved documents; these citations should be removed or replaced with relevant RAG-robustness literature.","section":"II-C, References [18]--[22]"},{"comment":"The cross-validation procedure is not described: the number of folds, the search range for tau, and the exact effectiveness metric used for selection are missing, which prevents reproduction.","section":"III-C, Figure 2"},{"comment":"The \"Step\" and \"Time\" metrics are said to be relative to single-step retrieval, but no formula or unit definition is provided; please specify how these values were computed.","section":"III-A"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the within-model ablation gives some reason to believe semantic entropy can help, but the main empirical evidence is compromised by the generator confound and by threshold fitting on the test benchmarks. I would support a major revision if the authors can provide model-controlled experiments, held-out threshold validation, and honest efficiency reporting. If those experiments are not feasible, the paper should be rejected. I also note that several references appear unrelated to the claims they support, which is a citation-hygiene concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper applies semantic entropy to decide when to retrieve in RAG, with a three-way split (no retrieval / single-step / multi-step). The idea is a natural extension of existing adaptive RAG work, and the paper is honest about some of its own limitations. But the main experimental comparison is not controlled: SUGAR runs on Llama-2-chat (7B), while the No Retrieval and Single-step baselines use FLAN-T5-XL. That alone makes Tables I and II uninterpretable as evidence about the retrieval policy. On top of that, the semantic entropy thresholds are selected by cross-validation on the very benchmarks used for the headline numbers, so the reported gains are partly fitted. The within-model ablation in Table III is the most informative part of the paper; it compares semantic-entropy-triggered retrieval against predictive-entropy-triggered retrieval on TriviaQA and SQuAD with the same generator, and it does show a consistent advantage for semantic entropy. That is a real, if small, contribution. The paper also claims the approach is 'not task- or data-dependent,' but the per-dataset threshold tuning contradicts that. It acknowledges the inference-time cost in Section III-C, though the abstract's 'efficient inference' is misleading. Bottom line: the core idea is plausible and the ablation is worth a look, but the central empirical claims need a controlled comparison with a fixed generator and thresholds tuned on held-out data before they can be taken seriously. Who is this for? Researchers working on adaptive RAG and uncertainty-based retrieval triggering. It's a useful pointer for the idea, but the experimental evidence is not strong enough to support the stated conclusions. I'd send it to review with a strong request for revision, mainly because the within-model ablation and the semantic-entropy application are worth examining more rigorously.","headline":"Plausible adaptive-RAG idea with a useful within-model ablation, but the main comparison is confounded by generator mismatch and threshold fitting.","tokens_in":8057,"tokens_out":2112,"would_cite":false,"duration_ms":19854,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims semantic entropy can decide when retrieval is needed, improving open-domain QA accuracy and efficiency without extra training.","keywords":["semantic entropy","adaptive retrieval","retrieval-augmented generation","question answering","uncertainty estimation","open-domain QA","selective retrieval","large language models"],"falsifier":"Run SUGAR's decision rule on a fixed generator (the same model for no-retrieval, single-step, predictive-entropy, and semantic-entropy conditions) over SQuAD and TriviaQA and compare accuracy and retrieval steps; if semantic-entropy gating does not beat predictive-entropy gating at matched thresholds, the claim that meaning-aware uncertainty drives the gains is falsified.","tokens_in":7225,"feed_emoji":"🔎","tokens_out":4170,"duration_ms":38923,"temperature":0.7,"pith_summary":"This paper argues that a language model's semantic uncertainty, measured as entropy over meaning-equivalent answer clusters, is a reliable signal for deciding when retrieval-augmented generation should be triggered. It introduces SUGAR, a training-free adaptive strategy that skips retrieval when the model is confident, performs one retrieval round at intermediate uncertainty, and performs multiple rounds at high uncertainty. On five open-domain question answering datasets, SUGAR reports higher accuracy than naive retrieval and prior adaptive RAG baselines, while often using fewer retrieval steps. The paper's significance is that uniform retrieval is wasteful and can even hurt answers by injecting noisy context, so a cheap, meaning-aware uncertainty gate could make RAG both more accurate and more efficient.","feed_headline":"Semantic entropy decides when RAG should retrieve","feed_subtitle":"Semantic uncertainty lets LLMs skip needless retrieval, preserving accuracy while using fewer retrieval steps on open-domain QA.","key_machinery":"The central mechanism is semantic entropy, defined as $SE(x) \\approx -|C|^{-1} \\sum_{i=1}^{|C|} \\log p(C_i \\mid x)$, where $C$ is a set of clusters of generated answers that share the same meaning, grouped by bidirectional entailment. Unlike token-level predictive entropy, it treats surface-form variations as one meaning and therefore separates genuine knowledge uncertainty from lexical choice. In SUGAR, this entropy is computed from the generator's own high-temperature samples and compared against thresholds that select among no retrieval, single-step retrieval, and multi-step retrieval, making the retrieval policy a direct function of the model's estimated knowledge boundary.","core_discovery":"The central claim is that semantic entropy, computed by sampling candidate answers and clustering them by bidirectional entailment, can serve as a task-independent controller for adaptive retrieval in open-domain QA. SUGAR first generates answers from the model's parametric knowledge alone; if semantic entropy is below a threshold, it answers directly, if intermediate it triggers single-step retrieval, and if high it triggers multi-step retrieval. The paper reports that this gate outperforms no retrieval, uniform single-step retrieval, and adaptive strategies based on entity popularity, reflection tokens, and question-complexity labels on SQuAD, Natural Questions, TriviaQA, HotpotQA, and 2WikiMultiHopQA, while reducing retrieval steps relative to multi-step IRCoT. The ablation further shows that semantic-entropy-gated retrieval beats predictive-entropy-gated retrieval on TriviaQA and SQuAD, supporting the claim that meaning-aware uncertainty mitigates overconfidence and lexical-form confusions.","pith_inferences":["Beyond QA, the same semantic-entropy gate could be applied to other knowledge-intensive generation tasks such as fact-checking or dialogue grounding, where knowing when to consult external sources is equally important; the paper does not test these settings.","Because SUGAR requires sampling multiple answers per query, its per-query overhead could be reduced by reusing samples across related queries or by approximating semantic clusters with a trained entailment model; the paper acknowledges the time cost but does not explore such optimizations.","A natural extension is to make the entropy thresholds adaptive per query or per dataset rather than fixed by cross-validation, which could improve robustness when the test distribution shifts away from the calibration set."],"forward_implications":["Adaptive retrieval can be performed without training a classifier or relying on task-specific labels, since the gate is computed from the generator's own outputs.","Semantic-entropy gating improves accuracy over both no-retrieval and uniform single-step retrieval on single-hop and multi-hop QA benchmarks, as reported in the paper.","The approach reduces retrieval steps relative to uniform multi-step retrieval methods like IRCoT, yielding more efficient inference on multi-hop questions.","The gate generalizes across datasets with different complexity levels, because the entropy computation does not depend on annotated complexity labels."],"supporting_citations":[{"why":"Supplies the semantic entropy computation (sampling, clustering by bidirectional entailment) that SUGAR adapts as its retrieval gate.","marker":"[29]"},{"why":"Adaptive-RAG baseline based on question complexity labels, the main adaptive method SUGAR compares against and argues its advantage over.","marker":"[10]"},{"why":"Self-RAG baseline using reflection tokens, a key comparison for adaptive retrieval behavior.","marker":"[23]"},{"why":"IRCoT multi-step retrieval baseline used in multi-hop comparisons and for measuring retrieval-step efficiency.","marker":"[26]"},{"why":"Adaptive Retrieval baseline based on entity popularity, providing another point of comparison for the uncertainty-gated approach.","marker":"[5]"},{"why":"Llama-2-chat (7B), the generator used in SUGAR and its ablation studies.","marker":"[38]"},{"why":"Contriever-MS MARCO, the dense retriever used to fetch external knowledge in SUGAR.","marker":"[7]"}],"fun_headline_variants":["Semantic uncertainty steers RAG retrieval steps","Retrieve less, not more: semantic entropy gate","Adaptive retrieval via semantic entropy in QA","SUGAR: Skip needless retrieval with semantic confidence","Know when to retrieve: semantic entropy does it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline comparison uses different base models for SUGAR and for the no-retrieval and single-step baselines; if the base model, not the retrieval decision, is what makes answers more accurate, the central claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Semantic uncertainty steers RAG retrieval steps","Retrieve less, not more: semantic entropy gate","Adaptive retrieval via semantic entropy in QA","SUGAR: Skip needless retrieval with semantic confidence","Know when to retrieve: semantic entropy does it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1321,"prompt_tokens":870,"completion_tokens":451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":378}},"tokens_in":486,"tokens_out":451,"duration_ms":4887,"temperature":1.0,"reasoning_tokens":378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:22:08.063199+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SUGAR's decision rule on a fixed generator (the same model for no-retrieval, single-step, predictive-entropy, and semantic-entropy conditions) over SQuAD and TriviaQA and compare accuracy and retrieval steps; if semantic-entropy gating does not beat predictive-entropy gating at matched thresholds, the claim that meaning-aware uncertainty drives the gains is falsified.","supporting_citations":[{"cited_title":"Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation,","cited_arxiv_id":null,"evidence_quote":"Supplies the semantic entropy computation (sampling, clustering by bidirectional entailment) that SUGAR adapts as its retrieval gate."},{"cited_title":"Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity,","cited_arxiv_id":null,"evidence_quote":"Adaptive-RAG baseline based on question complexity labels, the main adaptive method SUGAR compares against and argues its advantage over."},{"cited_title":"Self-rag: Learning to retrieve, generate, and critique through self-reflection,","cited_arxiv_id":null,"evidence_quote":"Self-RAG baseline using reflection tokens, a key comparison for adaptive retrieval behavior."},{"cited_title":"Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi- step questions,","cited_arxiv_id":null,"evidence_quote":"IRCoT multi-step retrieval baseline used in multi-hop comparisons and for measuring retrieval-step efficiency."},{"cited_title":"When not to trust language models: Investigating effectiveness of parametric and non-parametric memories,","cited_arxiv_id":null,"evidence_quote":"Adaptive Retrieval baseline based on entity popularity, providing another point of comparison for the uncertainty-gated approach."},{"cited_title":"Unsupervised dense information retrieval with con- trastive learning,","cited_arxiv_id":null,"evidence_quote":"Contriever-MS MARCO, the dense retriever used to fetch external knowledge in SUGAR."}],"review_version":1}