Beyond Self-Knowledge: Propagating Uncertainty Across Reasoning and Retrieval in LLMs

Chandan Kumar Sah; Li Zhang; Xiaoli Lian

REVIEW 3 major objections 4 minor 17 references

Beyond Self-Knowledge: Propagating Uncertainty Across Reasoning and Retrieval in LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A language model's own stated confidence can steer retrieval, improving answers while using fewer passages.

desk verdict Careful, honest empirical study of verbalized-confidence retrieval routing whose matched-random-control evidence for 'allocation value' doesn't match the method's own no-retrieval branch. read the letter →

arxiv 2607.25600 v1 pith:3AX3Z7ME submitted 2026-07-28 cs.IR cs.AIcs.CL

Chandan Kumar Sah , Xiaoli Lian , Li Zhang This is my paper

classification cs.IRcs.AIcs.CL

keywords retrievalroutingverbalizedconfidenceadaptiveblack-boxLLMuncertaintyretrieval-augmentedgenerationquestionansweringtokenefficiencycalibration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a black-box LLM's verbalized confidence estimate, obtained before any retrieval, can act as a routing signal for deciding when to fetch external evidence. It introduces a controller that asks the model for a structured provisional answer and a confidence score; if confidence falls below a validation-frozen threshold, the system retrieves top-5 passages and answers again, otherwise it returns the provisional answer directly. Across six QA benchmarks and three model families, this approach achieves the highest mean token-level F1 (0.483), beating always retrieval (0.467) and no retrieval (0.401) while reducing retrieved passages by 20.4%. A route-count-matched random control shows that the confidence-based selection of which questions to retrieve carries value in 17 of 18 settings, even though the confidence scores are poorly calibrated as absolute probabilities. The catch is that the extra probe raises total token usage by 28.2%, so the paper's claim is ultimately about smarter evidence allocation, not cheaper inference.

What carries the argument

The central mechanism is a black-box probe that returns a compact structured state—provisional answer, verbalized confidence c0, suggested action, and a short summary—followed by a frozen threshold router R(q,m)=1[c0<τm]. If the router is off, the system returns the provisional answer; if it is on, it runs TF-IDF top-5 retrieval and a second answer call. The confidence value is treated purely as an ordinal routing score, deliberately separated from absolute calibration, and the threshold is chosen on held-out validation data. This isolates the operational value of black-box confidence without needing logits, internal activations, or model fine-tuning.

What would settle it

Run a prospective experiment where the same probe and answer-call structure is used, but questions are routed to retrieval at the same per-cell rate by a random or hash-based rule instead of by confidence; if the random router matches BeyondUncertainty's F1 while the paper's post-hoc recombination does not, then the structured execution path, not confidence, drives the gain. Alternatively, compare the no-retrieval branch's provisional answers against plain no-retrieval outputs; if the probe prompt alone substantially improves F1, the allocation comparison is confounded.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that a model's stated confidence in a provisional answer is more useful as an ordinal ranking signal for retrieval allocation than as a calibrated probability of correctness. The paper shows that retrieving only for questions with confidence below a per-family threshold outperforms both always-retrieving and never-retrieving, and beats a random allocation that retrieves the exact same number of questions. The signal predicts question-level retrieval benefit only modestly (AUROC 0.628), so the paper's key claim is that verbalized confidence has decision utility, not calibration utility, for adaptive retrieval.

Load-bearing premise

The load-bearing premise is that the post-hoc route-count-matched random control is a valid counterfactual for what confidence routing would have done; if the probe's structured prompt and two-call execution change answer quality independently of which questions are routed, the 0.024 F1 allocation advantage would not show allocation value.

Editorial extensions

If this is right

Adaptive retrieval can be driven by a purely black-box confidence signal, requiring no access to hidden model states.
Selective retrieval based on low confidence improves mean F1 over both always retrieval and no retrieval while cutting retrieved passages by 20.4%.
The value comes from choosing which questions to retrieve, not merely from retrieving fewer: confidence routing beats a route-count-matched random policy in 17 of 18 settings.
Retrieval is not uniformly beneficial; in some cases it lowers answer quality, and confidence routing recovers a gain there.
The current probe makes the system retrieval-saving but not token-saving; total token usage rises 28.2%, so end-to-end efficiency gains require cheaper uncertainty estimators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A cheaper or free confidence signal (derived from sampling consistency, a shorter probe, or a single forward pass) could convert the demonstrated allocation gain into genuine token savings without losing the routing benefit.
Because the paper's route-count-matched control does not reproduce the probe's prompt and two-call execution, a directly executed randomized router with an identical probe and answer-call structure would be the sharpest test of whether the 0.024 F1 advantage is causal.
Combining verbalized confidence with other cheap signals, such as question complexity or retrieval score gaps, could close part of the 0.040 F1 gap between the method and the retrospective oracle.
Moving from controlled local corpora to a large open-domain index may change the routing decision because retrieval quality and distraction rates differ; testing this is a natural extension of the paper's controlled-conditions claim.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things up front. The experimental hygiene is genuinely good, and the headline claim about question-level allocation value is not actually established by the paper's key control.

What's new and useful: a 27,000-row evaluation of black-box verbalized-confidence routing across three model families and six QA benchmarks, with validation-frozen thresholds, Holm-corrected paired tests, failures retained in the denominator, and honest reporting of the token trade-off. The paper shows that the probe improves aggregate F1 (0.483 vs 0.467 always retrieval) while cutting retrieved passages by 20%, but costs 28.2% more tokens. The sensitivity analysis and the per-cell significance breakdown are refreshingly honest. Code and artifacts are released.

Now the soft spot. RQ3's route-count-matched random control is meant to show that confidence picks better questions than random at the same retrieval rate. But the control recombines static no-retrieval outputs for unselected questions, whereas BeyondUncertainty returns the structured probe's provisional answer. So the reported 0.024 F1 advantage and the 17-of-18 cell count could be driven by the probe's own answer being better than a plain no-retrieval answer on high-confidence questions, independent of which questions are routed. The paper explicitly concedes this in the limitations ('does not reproduce the structured probe, prompt path, or two-call execution'), yet Takeaway 3 still claims confidence carries allocation value. That's a mismatch. The fix is straightforward: the authors already have the a0 outputs; a matched random control that uses a0 for unselected questions would isolate allocation with zero additional API calls. Until that's reported, the causal reading of RQ3 is underdetermined.

The AUROC result (0.628) doesn't rescue it either: it measures prediction of static-policy benefit (never vs always retrieve), not the deployed benefit of the a0 branch. And even taking the paper's best face, the gains are modest: 0.016 F1 over always retrieval, with only 2 of 18 cells significantly favoring confidence after Holm correction.

Who should read it: people working on adaptive retrieval and uncertainty-based routing. The failure analysis, token accounting, and threshold sensitivity are useful references. It's a solid, honest paper that overreaches in one central claim. I'd send it to reviewers, but with a clear request: run the execution-matched control or soften Takeaway 3. The current version should be conditional, not accepted.

Referee Report

3 major / 4 minor

Summary. The paper proposes BeyondUncertainty, a black-box confidence-guided retrieval router for QA. A structured probe first elicits a provisional answer, a verbalized confidence score, and a short state summary; a validation-frozen, model-specific threshold then decides whether to return that provisional answer directly or retrieve top-5 TF-IDF passages and issue a second answer call. The authors evaluate three policies (no retrieval, always retrieve, confidence routing) on six QA benchmarks and three model families, yielding 9,000 paired question–model instances and 27,000 policy rows. They report mean token-level F1, retrieved passages, gateway-reported token usage, failure retention, Holm-corrected paired tests, calibration/ECE, AUROC for retrieval-benefit prediction, and a route-count-matched hash-based random control. The headline results are that confidence routing reaches 0.483 mean F1 versus 0.467 for always retrieval and 0.401 for no retrieval, reduces retrieved passages by 20.4%, and beats the matched random control in 17 of 18 dataset–model cells by an average of 0.024 F1, while increasing total token usage by 28.2%.

Significance. This is a carefully designed empirical study. The authors freeze validation thresholds before test evaluation, retain API and parsing failures in the denominator, apply one joint Holm correction across all 54 paired comparisons, include offline sensitivity analysis, and release the full evaluation artifact. If the allocation-value claim were cleanly supported, the paper would make a useful practical point: verbalized confidence, despite being poorly calibrated, can act as an ordinal black-box signal for retrieval routing. The main weakness, discussed below, is that the key RQ3 comparison does not isolate confidence-based allocation from the probe's execution path, and the RQ1 benefit definition uses the wrong no-retrieval baseline for the deployed system. Both issues are fixable with outputs the authors already possess. The descriptive aggregate quality/cost results are well supported by the reported tables.

major comments (3)

[Experimental Design — Route-count-matched and oracle controls; Results — RQ3; Takeaway 3] The route-count-matched random control is execution-mismatched in a way that is load-bearing for the central allocation claim. For unselected questions, BeyondUncertainty (S) returns the structured probe answer a0, whereas the random control returns the static no-retrieval output N. If R is the confidence-selected set and R' the hash-matched set, the reported S − random difference can be written as [Σ_R(F1_A − F1_N) − Σ_{R'}(F1_A − F1_N)]/N + Σ_{not R}(F1_{a0} − F1_N)/N. The final term is an unmeasured prompt effect: asking for JSON/confidence/summary may change answer quality conditional on high confidence, independent of which questions are routed. The paper itself concedes the control 'does not reproduce BeyondUncertainty's structured probe, prompt path, or two-call execution,' yet Takeaway 3 and the abstract still assert that confidence carries question-level allocation value. Becaus
[Results — RQ1, Eq. (5)] The RQ1 predictive evidence targets the wrong no-retrieval baseline. Eq. (5) defines Δret = F1_A − F1_N, and the paper reports AUROC = 0.628 for predicting Δret > 0. But the deployed router's no-retrieval branch is not N; it is the structured probe answer a0. The benefit that matters for routing is F1_A − F1_{a0}. Since a0 is available for all rows, the authors should report AUROC, rank correlation, and the matched-random decomposition for this deployed benefit. As written, RQ1 measures correlation with a static-policy benefit that the system never actually uses, and therefore cannot substitute for the missing execution-matched RQ3 control.
[Figure 4 and RQ3 text] The '17 of 18 cells' claim is based on a single deterministic SHA-256 ordering. 'Outperforms random allocation' is a distributional statement, and one hash is one draw from that distribution. No confidence intervals, multiple seeds, or permutation tests over the paired static outputs are reported. Since the control recombines existing outputs, resampling is free; a small set of random seeds or a paired permutation test would show whether the per-cell advantages are stable. Without this, the 17/18 count is not statistically grounded even setting aside the execution-path confound.

minor comments (4)

[Table 3 and Figure 3] Please state explicitly which policy's F1 is used for ρ(c0, F1) and for the binned 'final answer quality' in Figure 3. The text suggests no-retrieval/provisional-answer quality, but the table caption does not say.
[Figure 5, right panel] The right panel is an offline proxy constructed by recombining paired static outputs, not an executed policy. The caption should say 'offline proxy F1' on the axis or in the title, not only in the body text, to prevent readers from misreading it as executed policy results.
[Abstract and Experimental Design] '27,000 policy instances' is actually 3,000 questions × 3 models × 3 policies = 27,000 policy rows. Consider using 'policy rows' or 'policy executions' to avoid confusing instances with unique questions.
[Model families and decoding] The OpenAI endpoint uses the gateway default temperature because temperature 0 is rejected. Since the paper explicitly says bitwise determinism is not assumed, this is acceptable, but it should be listed among the reproducibility limitations in the Discussion as well.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: validation-frozen threshold, post-hoc matched control, and explicit execution-path limitation.

full rationale

The paper's derivation chain is self-contained and contains no step in which a claimed result reduces by construction to a fitted value or to a self-citation. The only fitted constant is the model-specific routing threshold τ_m (Table 1), selected on held-out validation data and frozen before test evaluation; the headline F1 comparisons are test-set evaluations. RQ1's AUROC 0.628 is a rank-association measure between the elicited verbalized confidence and observed static-policy benefit Δ_ret = F1_A − F1_N; no parameter is fitted to those test outcomes. RQ3's route-count-matched random control is a deterministic post-hoc recombination of paired static-policy outputs, not a fit of the reported 0.024 gain; the paper explicitly acknowledges that the control "does not reproduce BeyondUncertainty's structured probe, prompt path, or two-call execution" (RQ3 section) and frames the result as allocation value rather than a fully controlled causal effect. That is a validity limitation, not circularity: the comparison is not equal to its inputs by construction. No load-bearing self-citation appears; the reference to Ou et al. 2026 in Related Work is contextual, and if one of its authors overlaps with the present authors, it does not carry the argument. The oracle control is an upper bound, not a prediction. Therefore the score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are postulated. The router is a deterministic function of the model's own confidence and a fitted threshold; the only fitted quantities are the per-model threshold and the hand-set retrieval depth.

free parameters (2)

routing threshold τ_m = OpenAI 1.0; Gemini 1.0; Claude 0.9
Selected per model on held-out validation F1 before test; controls the retrieval/direct-answer trade-off and is the main fitted constant.
retrieval depth K = 5
Top-5 passages chosen by design (hand-set hyperparameter), applied to all policies.

assumptions (5)

domain assumption Verbalized confidence c0 is a stable ordinal signal for the same model across test questions.
The paper evaluates this empirically (AUROC 0.588-0.633) but does not prove stability across prompts or time.
domain assumption The dataset-specific TF-IDF corpus is a meaningful retrieval environment for the QA benchmarks.
The paper limits claims to controlled local corpora and acknowledges it is not a production-scale index.
domain assumption The route-count-matched random control recombining static outputs is a valid counterfactual for confidence routing.
The paper explicitly notes the control does not reproduce the probe or two-call execution path; if false, the 0.024 F1 gain may reflect prompt effects rather than allocation.
domain assumption Gateway-reported model aliases and token counts are accurate enough for comparison.
The paper states aliases are exactly as returned by the gateway and not independently verified; token usage is gateway-reported.
standard math Paired Wilcoxon signed-rank tests with one joint Holm correction are appropriate for the 54 cell comparisons.
Standard nonparametric paired testing used for matched per-cell comparisons.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond Self-Knowledge: Propagating Uncertainty Across Reasoning and Retrieval in LLMs." pith.science (2026). https://pith.science/paper/3AX3Z7ME

@misc{pith2026260725600,
  author       = {Pith},
  title        = {Pith review of: Beyond Self-Knowledge: Propagating Uncertainty Across Reasoning and Retrieval in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3AX3Z7ME}},
  note         = {Machine review of arXiv:2607.25600}
}

abstract

Retrieval-augmented generation improves knowledge-intensive question answering, but indiscriminate retrieval can introduce irrelevant evidence and unnecessary computation. We investigate whether verbalized confidence from black-box language models can serve as an actionable signal for retrieval routing. Our method, BeyondUncertainty, first elicits a structured provisional answer and confidence estimate, then applies a model-specific threshold selected on held-out validation data and frozen before test evaluation. Low-confidence questions receive top-$5$ TF--IDF retrieval followed by a second answer call, whereas high-confidence questions return the provisional answer directly. We evaluate 27,000 policy instances across six QA benchmarks, three model families, and three retrieval policies. BeyondUncertainty achieves $0.483$ mean token-level F1, compared with $0.467$ for always retrieval and $0.401$ for no retrieval, while reducing retrieved passages by $20.4\%$ relative to always retrieval. When matched on the number of questions routed to retrieval within each dataset--model cell, it outperforms a post-hoc random allocation in 17 of 18 settings, with an average gain of 0.024 F1. Although poorly calibrated as an absolute probability, probe uncertainty modestly predicts question-level retrieval benefit (AUROC $=0.628$). However, the additional probe increases total token usage by $28.2\%$, revealing a trade-off between more selective evidence acquisition and end-to-end token efficiency.

Figures

Figures reproduced from arXiv: 2607.25600 by the authors.

**Figure 1.** Implemented BeyondUncertainty pipeline. A structured black-box probe produces a provisional answer and confidence. A validation-frozen, model-specific threshold routes the question either directly to that answer or through top-5 sparse retrieval and a second answer call. Model family Gateway-reported alias τm OpenAI gpt-5.5 1.0 Gemini gemini-3.5-flash 1.0 Claude claude-sonnet-5 0.9 [PITH_FULL_IMAGE:figures/full_fi… view at source ↗

**Figure 2.** Aggregate answer quality and resource usage across 27,000 policy instances. Confidence routing achieves the highest [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Retrieval rate and final F1 across probe-confidence [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: The only exception is MuSiQue–OpenAI, for which [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 4.** Figure 4: Per-cell F1 difference between confidence routing [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Sensitivity and efficiency analyses. Left: Mean F1 versus gateway-reported token usage for each model family and retrieval policy. Right: Offline sensitivity to the routing threshold, constructed by recombining paired no-retrieval and alwaysretrieval outputs. Increasi…

**Figure 6.** Figure 6: Coverage and retained failures across datasets. Suc [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 4 linked inside Pith

[5]

1601–1611

Triviaqa:Alargescaledistantlysupervisedchallengedataset for reading comprehension. 1601–1611. Kadavath, S.; Conerly, T.; Askell, A.; Henighan, T.; Drain, D.; Perez, E.; Schiefer, N.; Hatfield-Dodds, Z.; DasSarma, N.;Tran-Johnson,E.;etal.2022. Languagemodels(mostly) know what they know.arXiv preprint arXiv:2207.05221. Kuhn, L.; Gal, Y.; and Farquhar, S

arXiv 2022
[6]

Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; et al

Seman- tic uncertainty: Linguistic invariances for uncertainty es- timation in natural language generation.arXiv preprint arXiv:2302.09664. Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; et al

arXiv
[8]

Mallen, A.; Asai, A.; Zhong, V.; Das, R.; Khashabi, D.; andHajishirzi,H.2023

Generating with con- fidence: Uncertainty quantification for black-box large lan- guage models.arXiv preprint arXiv:2305.19187. Mallen, A.; Asai, A.; Zhong, V.; Das, R.; Khashabi, D.; andHajishirzi,H.2023. Whennottotrustlanguagemodels: Investigatingeffectivenessofparametricandnon-parametric memories. InProceedings of the 61st annual meeting of the associa...

arXiv 2023
[9]

InProceedings of the 2023 conference on empirical methods in natural language pro- cessing, 9004–9017

Selfcheckgpt: Zero-resource black-box hallucination detection for gener- ative large language models. InProceedings of the 2023 conference on empirical methods in natural language pro- cessing, 9004–9017. Mielke,S.J.;Szlam,A.;Dinan,E.;andBoureau,Y.-L.2022. Reducingconversationalagents’overconfidencethroughlin- guisticcalibration.TransactionsoftheAssociati...

2023
[12]

In Proceedings of the 62nd Annual Meeting of the Associa- tionforComputationalLinguistics(Volume1:LongPapers), 12991–13013

Dra- gin: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Proceedings of the 62nd Annual Meeting of the Associa- tionforComputationalLinguistics(Volume1:LongPapers), 12991–13013. Tian, K.; Mitchell, E.; Zhou, A.; Sharma, A.; Rafailov, R.; Yao,H.;Finn,C.;andManning,C.D.2023. Justaskforcal- ibrati...

2023
[14]

2024: 23650– 23678

Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. 2024: 23650– 23678. Xu, F.; Shi, W.; and Choi, E

2024
[15]

Yang,Z.;Qi,P.;Zhang,S.;Bengio,Y.;Cohen,W.;Salakhut- dinov, R.; and Manning, C

Recomp: Improving retrieval-augmentedlmswithcompressionandselectiveaug- mentation.arXiv preprint arXiv:2310.04408. Yang,Z.;Qi,P.;Zhang,S.;Bengio,Y.;Cohen,W.;Salakhut- dinov, R.; and Manning, C. D

arXiv
[16]

InProceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), 27022–27043

Seakr: Self-aware knowledge retrieval foradaptiveretrievalaugmentedgeneration. InProceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), 27022–27043. Yoran,O.;Wolfson,T.;Ram,O.;andBerant,J.2024.Making Retrieval-AugmentedLanguageModelsRobusttoIrrelevant Context. InTheTwelfthInternationalConferenceo...

2024

Show all 17 references

[17]

InProceed- ings of the 2024 Joint International Conference on Com- putational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 5191–5201

Does ChatGPT know that it does not know? Eval- uating the Black-Box Calibration of ChatGPT. InProceed- ings of the 2024 Joint International Conference on Com- putational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 5191–5201

2024
[2016]

InProceedings of the 2016 conference on empirical methods in natural language processing, 2383–2392

Squad: 100,000+ questions for machine comprehension of text. InProceedings of the 2016 conference on empirical methods in natural language processing, 2383–2392. Shi,F.;Chen,X.;Misra,K.;Scales,N.;Dohan,D.;Chi,E.H.; Schärli, N.; and Zhou, D

2016
[2017]

InInternational conference on machine learning, 1321–1330

On calibration of modern neural networks. InInternational conference on machine learning, 1321–1330. PMLR. Ho, X.; Nguyen, A.-K. D.; Sugawara, S.; and Aizawa, A. 2020.Constructingamulti-hopqadatasetforcomprehensive evaluation of reasoning steps. 6609–6625. Holm, S

2020
[2022]

Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A.2023

MuSiQue: Multihop Questions via Single-hop Question Composition.Transactions of the Association for Computational Linguistics, 10: 539–554. Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A.2023. Interleavingretrievalwithchain-of-thoughtreason- ing for knowledge-int...

2023
[2023]

InProceedings of the 2023 confer- ence on empirical methods in natural language processing, 7969–7992

Active retrieval augmented generation. InProceedings of the 2023 confer- ence on empirical methods in natural language processing, 7969–7992. Joshi, M.; Choi, E.; Weld, D. S.; and Zettlemoyer, L

2023
[2024]

2024: 9112–9141

Self-rag:Learningtoretrieve,generate,andcritiquethrough self-reflection. 2024: 9112–9141. Farquhar,S.;Kossen,J.;Kuhn,L.;andGal,Y.2024. Detect- ing hallucinations in large language models using semantic entropy.Nature, 630(8017): 625–630. Guo, C.; Pleiss, G.; Sun, Y.; and Weinb...

2024
[2025]

InFindings of the Association for Computational Linguistics: EMNLP 2025, 16979–16994

Semantic Contribution-Aware Adaptive Retrieval for Black-Box Models. InFindings of the Association for Computational Linguistics: EMNLP 2025, 16979–16994. Lin, Z.; Trivedi, S.; and Sun, J

2025
[2026]

InFind- ings of the Association for Computational Linguistics: ACL 2026, 446–465

Browseconf: Confidence-guidedtest-timescalingforwebagents. InFind- ings of the Association for Computational Linguistics: ACL 2026, 446–465. Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P

2026
[7050]

Howcan weknowwhenlanguagemodelsknow?onthecalibrationof languagemodelsforquestionanswering.Transactionsofthe Association for Computational Linguistics, 9: 962–977

Jiang,Z.;Araki,J.;Ding,H.;andNeubig,G.2021. Howcan weknowwhenlanguagemodelsknow?onthecalibrationof languagemodelsforquestionanswering.Transactionsofthe Association for Computational Linguistics, 9: 962–977. Jiang,Z.;Xu,F.F.;Gao,L.;Sun,Z.;Liu,Q.;Dwivedi-Yu,J.; Yang, Y.; Callan,...

2021

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [5]

1601–1611

Triviaqa:Alargescaledistantlysupervisedchallengedataset for reading comprehension. 1601–1611. Kadavath, S.; Conerly, T.; Askell, A.; Henighan, T.; Drain, D.; Perez, E.; Schiefer, N.; Hatfield-Dodds, Z.; DasSarma, N.;Tran-Johnson,E.;etal.2022. Languagemodels(mostly) know what they know.arXiv preprint arXiv:2207.05221. Kuhn, L.; Gal, Y.; and Farquhar, S

arXiv 2022

[2] [6]

Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; et al

Seman- tic uncertainty: Linguistic invariances for uncertainty es- timation in natural language generation.arXiv preprint arXiv:2302.09664. Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; et al

arXiv

[3] [8]

Mallen, A.; Asai, A.; Zhong, V.; Das, R.; Khashabi, D.; andHajishirzi,H.2023

Generating with con- fidence: Uncertainty quantification for black-box large lan- guage models.arXiv preprint arXiv:2305.19187. Mallen, A.; Asai, A.; Zhong, V.; Das, R.; Khashabi, D.; andHajishirzi,H.2023. Whennottotrustlanguagemodels: Investigatingeffectivenessofparametricandnon-parametric memories. InProceedings of the 61st annual meeting of the associa...

arXiv 2023

[4] [9]

InProceedings of the 2023 conference on empirical methods in natural language pro- cessing, 9004–9017

Selfcheckgpt: Zero-resource black-box hallucination detection for gener- ative large language models. InProceedings of the 2023 conference on empirical methods in natural language pro- cessing, 9004–9017. Mielke,S.J.;Szlam,A.;Dinan,E.;andBoureau,Y.-L.2022. Reducingconversationalagents’overconfidencethroughlin- guisticcalibration.TransactionsoftheAssociati...

2023

[5] [12]

In Proceedings of the 62nd Annual Meeting of the Associa- tionforComputationalLinguistics(Volume1:LongPapers), 12991–13013

Dra- gin: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Proceedings of the 62nd Annual Meeting of the Associa- tionforComputationalLinguistics(Volume1:LongPapers), 12991–13013. Tian, K.; Mitchell, E.; Zhou, A.; Sharma, A.; Rafailov, R.; Yao,H.;Finn,C.;andManning,C.D.2023. Justaskforcal- ibrati...

2023

[6] [14]

2024: 23650– 23678

Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. 2024: 23650– 23678. Xu, F.; Shi, W.; and Choi, E

2024

[7] [15]

Yang,Z.;Qi,P.;Zhang,S.;Bengio,Y.;Cohen,W.;Salakhut- dinov, R.; and Manning, C

Recomp: Improving retrieval-augmentedlmswithcompressionandselectiveaug- mentation.arXiv preprint arXiv:2310.04408. Yang,Z.;Qi,P.;Zhang,S.;Bengio,Y.;Cohen,W.;Salakhut- dinov, R.; and Manning, C. D

arXiv

[8] [16]

InProceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), 27022–27043

Seakr: Self-aware knowledge retrieval foradaptiveretrievalaugmentedgeneration. InProceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), 27022–27043. Yoran,O.;Wolfson,T.;Ram,O.;andBerant,J.2024.Making Retrieval-AugmentedLanguageModelsRobusttoIrrelevant Context. InTheTwelfthInternationalConferenceo...

2024