Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UProp argues that a language model's uncertainty during multi-step decisions splits into intrinsic uncertainty about the current step and extrinsic uncertainty inherited from earlier decisions, and that tracking the inherited part…

desk verdict Useful empirical work on propagating uncertainty through LLM agent trajectories, but the main convergence theorem does not hold as stated. read the letter →

arxiv 2506.17419 v1 pith:X7J73GID submitted 2025-06-20 cs.CL cs.AIcs.LGstat.ML

Jinhao Duan , James Diffenderfer , Sandeep Madireddy , Tianlong Chen , Bhavya Kailkhura , Kaidi Xu This is my paper · ORCID
classification cs.CLcs.AIcs.LGstat.ML
keywords uncertaintyquantificationlargelanguagemodelsmulti-stepdecision-makingLLMagentsmutualinformationpointwisepropagationhallucinationdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to answer when an LLM should be trusted as it makes several decisions in a row. Its claim is that the uncertainty of the decision at step t is the sum of two pieces: how unsure the model is about the current step given all prior steps, and how much uncertainty is inherited from earlier decisions because those earlier choices could have gone differently. Existing single-turn uncertainty quantification targets only the first piece, so the paper introduces UProp to estimate the second piece as a pointwise mutual information over sampled decision trajectories. On AgentBench-OS, HotpotQA, and StrategyQA with several LLMs, adding this inherited-uncertainty term to a standard entropy score raises AUROC for detecting wrong agent outputs by 2.3% to 11% over single-turn baselines. If correct, this gives a practical information-theoretic signal for deciding when to stop or double-check a multi-step agent.

What carries the argument

The load-bearing object is the trajectory uncertainty decomposition $H(y_t|x) = H(y_t|y_{1:t-1},x) + \sum_{i<t} I(y_t; y_i | y_{i+1:t-1},x)$. The estimator UProp is built on Trajectory-Dependent Decision Process (TDP) sampling—at each step, sample $N$ decisions conditioned on the realized prefix, then keep one sampled decision as the prefix for the next step—which converts the exponential mutual-information integral into a linear number of conditional samples. Extrinsic uncertainty is then approximated by dPMI, which reduces to $-\log$ of a Gaussian-kernel-weighted count of how many sampled previous decisions resemble the realized one; string fuzzy matching over the agent's action gives the distance. Two convergence theorems (for TDP averaging and for the kernel PMI) are what license the approximation, with the second resting on a local smoothness assumption on $p_\theta(y_t | y_{t-1}, x)$.

What would settle it

Enumerate a toy two-step decision problem with a tiny action vocabulary, compute the true pointwise mutual information exactly by summing over all pairs of decisions, and compare it with UProp's dPMI at growing $N$ and $Z$; if the gap does not shrink to zero for appropriately shaped kernels with fixed sharpness $\tau$, the convergence claim fails. A second check: run UProp on two trajectories that reach the same observation through very different preceding actions; if the score moves with the surface similarity of the actions rather than with the information state, the kernel is measuring text distance, not inherited uncertainty.

Watch

Extended reading notes

Core claim

The paper's central claim is that the total uncertainty $H(y_t|x)$ of the decision at step $t$ in a multi-step agent decomposes exactly into $H(y_t|y_{1:t-1},x)$ plus a cumulative sum of conditional mutual informations $I(y_t; y_i | y_{i+1:t-1},x)$, so the "extrinsic" uncertainty inherited from every earlier decision is a well-defined information-theoretic quantity. Because direct mutual information requires exploring an exponentially branching decision space, UProp samples Trajectory-Dependent Decision Processes: at each step it draws $N$ candidate decisions, keeps one sampled decision as the realized prefix, and continues; each TDP is one linear trajectory plus $N$ per-step samples. On each TDP, the estimator replaces mutual information with pointwise mutual information and further reduces it to $dPMI(y_t; y_{t-1}^{(k)} | x) = -\log \sum_n \kappa_N(d(y_{t-1}^{(n)}, y_{t-1}^{(k)}))$, a kernel-weighted count of alternative previous decisions similar to the realized one. The paper presents convergence theorems for the TDP average and for the kernel PMI under a local smoothness assumption, and reports that ablating the extrinsic term degrades AUROC more than ablating the intrinsic term.

Load-bearing premise

The load-bearing premise is that the model's next-step distribution changes smoothly as the previous decision varies, so samples drawn from one decision path can stand in for samples from all possible paths; if that mismatch is not covered by the smoothness assumption, the kernel estimate is not actually approximating the defined mutual information.

Editorial extensions

If this is right

  • The full UProp score, which sums a per-step entropy with the extrinsic term, improves AUROC for detecting wrong answers on AgentBench-OS and StrategyQA across GPT-4.1-Nano, GPT-3.5-Turbo, Gemma-2-27b-it, DeepSeek-V3, and Qwen2.5-72B-Instruct.
  • The extrinsic component carries more of the signal than the intrinsic component: removing it from UProp lowers AUROC more than removing the intrinsic component does, so single-turn uncertainty scores alone under-use the trajectory's history.
  • UProp keeps most of its advantage when the numbers of trajectories and of per-step samples are as low as 2-4, so the estimate remains feasible when environment calls are expensive.
  • The same uncertainty score can serve as a rejection rule and as a selector of the most reliable answer among multiple generations, improving success rate over unguided decoding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because dPMI depends only on sampled actions and a string-similarity kernel, it should transfer to agents whose actions are structured function calls; if it fails there, the action-distance choice is the bottleneck rather than the information decomposition.
  • Editorial extension: the paper's framework measures total uncertainty but does not separate model-epistemic from environment-aleatoric uncertainty; a natural follow-up would be to extend the same mutual-information decomposition to distinguish the two.
  • Editorial extension: the kernel sharpness is a hyperparameter, so the testable claim is whether the reported AUROC gains survive across a range of kernel sharpness values without tuning on each benchmark.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces UProp, an uncertainty quantification method for LLM multi-step agentic decision-making. It decomposes the total uncertainty at each decision step into an intrinsic component, estimated by single-step predictive entropy, and an extrinsic component, defined as a mutual-information quantity between the current decision and preceding decisions. UProp estimates the extrinsic component by sampling multiple trajectory-dependent decision processes (TDPs), computing a pointwise mutual information (PMI) score for each trajectory, and aggregating these scores with a step-length normalization. The authors provide two convergence theorems, one for TDP sampling (Theorem 1) and one for the PMI approximation (Theorem 2), and evaluate the method on AgentBench-OS, HotpotQA, and StrategyQA with GPT-4.1-Nano, GPT-3.5-Turbo, Gemma-2-27b-it, DeepSeek-V3, and Qwen2.5-72B-Instruct. The paper reports AUROC improvements of 2.3% to 11% over single-turn uncertainty baselines and includes ablations, sampling-efficiency studies, and selective-prediction experiments.

Significance. The conceptual decomposition of decision uncertainty into intrinsic and extrinsic components is appealing, and the TDP sampling protocol is a practical way to address the exponential decision space in multi-step LLM reasoning. The empirical evaluation is broad in terms of benchmarks, backbone LLMs, and baselines, and the paper explicitly studies sampling efficiency and ablation of the two uncertainty components. If the theoretical claims were established, the paper would make a solid contribution to LLM uncertainty quantification. However, the current proof of Theorem 2 has a distribution mismatch that undermines the claim that the proposed dPMI estimator converges to the defined PMI, and the reported empirical gains lack error bars and significance tests. The method may still be useful as a heuristic score, but the paper's central theoretical justification is not yet established.

major comments (4)
  1. [Section A.5, Eq. (18)] The proof of Theorem 2 replaces an integral against the marginal p(y_{t-1}|x) with a Monte Carlo average over TDP samples y_{t-1}^{(i)}, but those samples are drawn from the conditional distribution p_θ(y_{t-1}|y_{1:t-2}^{(k)},x) of a single trajectory prefix. These are generally different distributions, and the local-smoothness assumption on p_θ(y_t|·,x) does not control the discrepancy between the conditional and marginal distributions of y_{t-1}. The omitted mass of the marginal outside the kernel neighborhood is therefore uncontrolled, and Theorem 2 is not established as stated. The authors need either a sampling scheme that targets the marginal, or an additional argument that the conditional samples are representative of the marginal with a quantitative bound on the approximation error.
  2. [Section 3.3, Eq. (8)] Substituting Eq. (7) into Eq. (6) causes the conditional likelihood p_θ(y_t|y_{t-1}^{(k)},x) to factor out of the numerator and denominator, so dPMI reduces to -log Σ_n K_N(d(y_{t-1}^{(n)}, y_{t-1}^{(k)})), which is independent of y_t. This is consistent with the local-smoothness approximation only if the log-ratio in the PMI is treated as constant in y_t, but the paper does not state this explicitly. As written, Eq. (8) measures the local density of the preceding action under the TDP samples rather than the mutual information defined in Eq. (4). The authors should either present dPMI as a heuristic proxy for extrinsic uncertainty, weakening the wording around Theorem 2, or derive a corrected estimator that retains dependence on y_t.
  3. [Tables 1-4 and Figure 2] No confidence intervals, standard deviations, or significance tests are reported for the AUROC and AUARC results. Several reported differences are small relative to the likely sampling variability, for example StrategyQA GPT-3.5-Turbo in Table 1 (UProp 0.604 vs. SE 0.623), and the paper's claim of significant outperformance by 2.3% to 11% AUROC is not supported without error bars or a paired statistical test. Please add uncertainty estimates and significance tests, such as bootstrapping or the DeLong test, for at least the main results in Tables 1, 2, and 4.
  4. [Section 3.3 and Section 4.1] The Gaussian kernel sharpness τ is a free hyperparameter that directly controls dPMI through K_N, but the paper reports no sensitivity analysis and does not state how τ was chosen or whether the same value was used across all benchmarks and models. Since the reported gains could depend on this choice, the authors should report AUROC for a range of τ values or provide a principled selection procedure.
minor comments (5)
  1. [Section A.4] In the last line of the proof of Theorem 1, the text says convergence holds 'with N→∞', but the statement of the theorem requires Z→∞; the proof also relies on Theorem 2 for the dPMI terms, so it is not independent of the PMI approximation.
  2. [Equations (7) and (18)] The index n is used ambiguously both as the summation index over TDP samples and as the label of the selected trajectory sample, and Eq. (18) contains typographical errors such as 'dist(yyt−1,y' and an unclosed parenthesis. Please clean up the notation and typos.
  3. [Tables 1 and 4] Several entries in the tables have formatting errors that make them hard to read, such as '0.6200.707' in Table 1 and '0.5510.506' in Table 4; please insert proper spacing or column separators.
  4. [Section 3.3, Step Length-Normalization] The normalization factor λ_z is defined using dPMI terms, and because the kernel weights in Eq. (8) are not normalized, dPMI can be negative if the kernel sum exceeds one. The paper should clarify whether extrinsic uncertainty can be negative and how such cases are handled in the total uncertainty estimate.
  5. [Section 4.1 and Appendix B] The paper says code will be released, but it does not provide the exact τ value, random seeds, or the number of sampled trajectories per question in the appendix; including these reproducibility details would strengthen the empirical claims.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: UProp's AUROC gains are externally benchmarked; the dPMI cancellation and TDP-sample mismatch are soundness gaps, not circular derivations.

full rationale

UProp's uncertainty decomposition (Eqs. 1-3) starts from standard chain-rule and entropy identities, so the framework is not defined in terms of the target AUROC result. No parameter is fitted to correctness labels or to the test benchmarks; the kernel sharpness tau is a hand-chosen hyperparameter and is not optimized against the evaluation metric. The baseline comparisons use external UQ methods and measured AUROC, so the empirical claim is self-contained rather than forced by construction. The paper's self-citations (SAR, GTBench) are used as baselines and related work, not as load-bearing justification for UProp's estimator, and no uniqueness theorem is imported from the authors' prior work. The two formal problems raised by a skeptical reading are real but are not circularity under the stated criteria: substituting Eq. (7) into Eq. (6) makes the conditional likelihood cancel, so Eq. (8) is exactly the negative log of a kernel-weighted sum over previous decisions and loses dependence on y_t, and Section A.5's Monte Carlo step samples from the TDP conditional prefix distribution rather than the marginal p(y_{t-1}|x). These are soundness/correctness gaps in Theorem 2, not cases where the paper fits a parameter to the target result or defines the conclusion into its premises. Accordingly, the circularity score is low.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central estimators rest on standard entropy identities plus three domain assumptions: deterministic transitions, local smoothness of the conditional decision distribution, and semantic meaningfulness of string-fuzzy distance. No new physical entities are introduced. The kernel sharpness τ and sampling budgets Z, N are hand-chosen hyperparameters, with τ unreported.

free parameters (2)
  • Gaussian kernel sharpness tau (τ) = unreported
    Kτ(x) = (1/sqrt(2π)e^{−x²/2})^τ; the paper states τ controls sharpness and says 'we take s=N', but never reports the value of τ used in experiments. This is a hand-chosen hyperparameter that directly determines how much surrounding samples influence dPMI.
  • TDP trajectory sample count Z and per-step sample count N = Z=10, N=10 (default)
    These are sampling budgets chosen by the authors for the main results. They affect estimator variance and performance (Figure 2), but they are not fitted to optimize AUROC; still they are design choices on which the central results depend.
assumptions (4)
  • standard math Chain rule of entropy and mutual information decomposition (Eq. 2, Section A.2), used to split total uncertainty into intrinsic and extrinsic parts.
    The decomposition H(yt|x) = H(yt|yt−1,x) + I(yt;yt−1|x) is a textbook identity from Cover (1999).
  • domain assumption The observation transition function is deterministic given the preceding decisions, so the decision distribution yt depends only on y1:t−1 and the instruction x (Section 2.2).
    If the environment is stochastic or partially observable, the conditioning set must include observations, and the stated decomposition changes.
  • ad hoc to paper Local smoothness of pθ(yt|yt−1,x) in yt−1 (Theorem 2, Section 3.3) is asserted as 'mild' and used to justify replacing the marginal p(yt|x) with a kernel-weighted average over conditional samples.
    The paper cites Malinovskii et al. (2024) for this assumption but provides no empirical check in action space. The convergence theorem depends on it.
  • ad hoc to paper The string fuzzy matching distance d between actions (thefuzz) is assumed to be a meaningful metric for semantic closeness of decisions such that the kernel K(d(·,·)) captures the local structure of the decision space (Section A.6).
    No validation is provided that fuzzy string similarity is aligned with the probabilistic smoothness of the LLM's decision distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making." pith.science (2026). https://pith.science/paper/X7J73GID

@misc{pith2026250617419,
  author       = {Pith},
  title        = {Pith review of: UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X7J73GID}},
  note         = {Machine review of arXiv:2506.17419}
}
read the original abstract

As Large Language Models (LLMs) are integrated into safety-critical applications involving sequential decision-making in the real world, it is essential to know when to trust LLM decisions. Existing LLM Uncertainty Quantification (UQ) methods are primarily designed for single-turn question-answering formats, resulting in multi-step decision-making scenarios, e.g., LLM agentic system, being underexplored. In this paper, we introduce a principled, information-theoretic framework that decomposes LLM sequential decision uncertainty into two parts: (i) internal uncertainty intrinsic to the current decision, which is focused on existing UQ methods, and (ii) extrinsic uncertainty, a Mutual-Information (MI) quantity describing how much uncertainty should be inherited from preceding decisions. We then propose UProp, an efficient and effective extrinsic uncertainty estimator that converts the direct estimation of MI to the estimation of Pointwise Mutual Information (PMI) over multiple Trajectory-Dependent Decision Processes (TDPs). UProp is evaluated over extensive multi-step decision-making benchmarks, e.g., AgentBench and HotpotQA, with state-of-the-art LLMs, e.g., GPT-4.1 and DeepSeek-V3. Experimental results demonstrate that UProp significantly outperforms existing single-turn UQ baselines equipped with thoughtful aggregation strategies. Moreover, we provide a comprehensive analysis of UProp, including sampling efficiency, potential applications, and intermediate uncertainty propagation, to demonstrate its effectiveness. Codes will be available at https://github.com/jinhaoduan/UProp.

Figures

Figures reproduced from arXiv: 2506.17419 by the authors.

Figure 1
Figure 1. Overall pipeline of uncertainty propagation of LLMs in multi-step agentic decision-making. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparing the sampling efficiency of UProp with baselines. Baselines We consider 7 popular single-step LLM UQ methods: Perplexity (PPL), Lexical Similarity (LS) Fomicheva et al. (2020), PE Malinin and Gales (2020), SE Kuhn et al. (2023), Deg Lin et al. (2024b), SD Qiu and Miikkulainen (2024), sentSAR Duan et al. (2024a). For a fair comparison and also a straightforward adaptation, baselines are calculated over the s… view at source ↗
Figure 3
Figure 3. Uncertainty as the correctness indicator [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The percentage of intrinsic and extrinsic uncertainty at each step. The Red shadow area is [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Detailed uncertainty percentage at each decision step. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Uncertainty Quantification for Retrieval-Augmented Reasoning

    cs.IR 2025-10 conditional novelty 6.0 of 10

    R2C perturbs reasoning states (paraphrasing, rethinking, validating) to score consistency, improving UQ AUROC by over 5% on average for retrieval-augmented reasoning.

  2. Runtime Uncertainty Monitoring for LLM-Based Multi-Agent Systems Using Bayesian Networks

    cs.AI 2026-07 conditional novelty 5.0 of 10

    A Bayesian-network monitor built on calibrated LLM log-probabilities gives workflow-level uncertainty scores for an actuarial multi-agent system, reproducing baseline RMSE but not clearly separating normal from pertur...

  3. LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems

    cs.AI 2025-12 reject novelty 4.0 of 10

    LEC proposes a +1-corrected threshold for FDR control in selective prediction and routing, but the finite-sample guarantee rests on a false exchangeability identity and is not valid for arbitrary exchangeable data.

Reference graph

Works this paper leans on

44 extracted references · 10 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Rest meets react: Self-improvement for multi-step reasoning llm agent

    Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, et al. Rest meets react: Self-improvement for multi-step reasoning llm agent. arXiv preprint arXiv:2312.10003, 2023

    arXiv 2023

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  5. [5]

    Inside: Llms' internal states retain the power of hallucination detection

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. Inside: Llms' internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744, 2024

    arXiv 2024

  6. [6]

    Elements of information theory

    Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999

    1999

  7. [7]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bing-Li Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dong-Li Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaji...

    arXiv 2024

  8. [8]

    A survey of embodied ai: From simulators to research tasks

    Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6 0 (2): 0 230--244, 2022

    2022

Show all 44 references
  1. [9]

    Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models

    Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the As...

    2024

  2. [10]

    Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations

    Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations. arXiv preprint arXiv:2402.12348, 2024 b

    2024 arXiv

  3. [11]

    Fomicheva, Shuo Sun, Lisa Yankovskaya, F

    M. Fomicheva, Shuo Sun, Lisa Yankovskaya, F. Blain, Francisco Guzm \'a n, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8: 0 539...

    2020

  4. [12]

    A survey of uncertainty in deep neural networks

    Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56 0 (Suppl 1): 0 151...

    2023

  5. [13]

    Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9: 0 346--361, 2021

    2021

  6. [14]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Informatio...

    2025

  7. [15]

    Towards mitigating hallucination in large language models via self-reflection

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating hallucination in large language models via self-reflection. arXiv preprint arXiv:2310.06271, 2023

    2023 arXiv

  8. [16]

    Estimating mutual information

    Alexander Kraskov, Harald St \"o gbauer, and Peter Grassberger. Estimating mutual information. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 69 0 (6): 0 066138, 2004

    2004

  9. [17]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023

    2023

  10. [18]

    Contextualized sequence likelihood: Enhanced confidence scores for natural language generation

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Contextualized sequence likelihood: Enhanced confidence scores for natural language generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10351--10368, 2024 a

    2024

  11. [19]

    Generating with confidence: Uncertainty quantification for black-box large language models

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. Transactions on Machine Learning Research, 2024 b

    2024

  12. [20]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023

    2023 arXiv

  13. [21]

    Aligning cyber space with physical world: A comprehensive survey on embodied ai

    Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. Aligning cyber space with physical world: A comprehensive survey on embodied ai. arXiv preprint arXiv:2407.06886, 2024

    2024 arXiv

  14. [22]

    Uncertainty estimation in deep learning with application to spoken language assessment

    Andrey Malinin. Uncertainty estimation in deep learning with application to spoken language assessment. PhD thesis, 2019

    2019

  15. [23]

    Uncertainty estimation in autoregressive structured prediction

    Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650, 2020

    2002 arXiv

  16. [24]

    Pv-tuning: Beyond straight-through estimation for extreme llm compression

    Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. Pv-tuning: Beyond straight-through estimation for extreme llm compression. Advances in Neural Information Processing Systems, 37: 0 5074--5121, 2024

    2024

  17. [25]

    Accuracy-rejection curves (arcs) for comparing classification methods with a reject option

    Malik Sajjad Ahmed Nadeem, Jean-Daniel Zucker, and Blaise Hanczar. Accuracy-rejection curves (arcs) for comparing classification methods with a reject option. In International Workshop on Machine Learning in Systems Biology, 2009

    2009

  18. [26]

    Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities

    Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities. Advances in Neural Information Processing Systems, 37: 0 8901--8929, 2024

    2024

  19. [27]

    Art: Automatic multi-step reasoning and tool-use for large language models

    Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014, 2023

    2023 arXiv

  20. [28]

    Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space

    Xin Qiu and Risto Miikkulainen. Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space. arXiv preprint arXiv:2405.13845, 2024

    2024 arXiv

  21. [29]

    Welty, Christopher A

    Gemma Team Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L'eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram'e, Johan Ferret, Peter Liu, Pouya Dehghani Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charl...

    2024 arXiv

  22. [30]

    thefuzz: Fuzzy string matching in python

    SeatGeek. thefuzz: Fuzzy string matching in python. https://pypi.org/project/thefuzz/, 2020. Accessed: May 14, 2025

    2020

  23. [31]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 8634--8652, 2023

    2023

  24. [32]

    Q*: Improving multi-step reasoning for llms with deliberative planning

    Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q*: Improving multi-step reasoning for llms with deliberative planning. arXiv preprint arXiv:2406.14283, 2024 a

    2024 arXiv

  25. [33]

    A survey on large language model based autonomous agents

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18 0 (6): 0 186345, 2024 b

    2024

  26. [34]

    Stepwise informativeness search for improving llm reasoning

    Siyuan Wang, Enda Zhao, Zhongyu Wei, and Xiang Ren. Stepwise informativeness search for improving llm reasoning. arXiv preprint arXiv:2502.15335, 2025

    2025 arXiv

  27. [35]

    Graph-augmented reasoning: Evolving step-by-step knowledge graph retrieval for llm reasoning

    Wenjie Wu, Yongcheng Jing, Yingjie Wang, Wenbin Hu, and Dacheng Tao. Graph-augmented reasoning: Evolving step-by-step knowledge graph retrieval for llm reasoning. arXiv preprint arXiv:2503.01642, 2025

    2025 arXiv

  28. [36]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  29. [37]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018

    2018 arXiv

  30. [38]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  31. [39]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  32. [40]

    Reasoning in flux: Enhancing large language models reasoning through uncertainty-aware adaptive guidance

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Zhiyuan Zeng, Xiaonan Li, Junqi Dai, Qinyuan Cheng, Xuan-Jing Huang, and Xipeng Qiu. Reasoning in flux: Enhancing large language models reasoning through uncertainty-aware adaptive guidance. In Proceedings of the 62nd Annual Meeting of the...

    2024

  33. [41]

    Large language models for robotics: A survey

    Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S Yu. Large language models for robotics: A survey. arXiv preprint arXiv:2311.07226, 2023

    2023

  34. [42]

    Saup: Situation awareness uncertainty propagation on llm agent

    Qiwei Zhao, Xujiang Zhao, Yanchi Liu, Nayeon Lee, Etsuko Ishii, Wei Fung, PascaleCheng, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Huaxiu Yao, and Haifeng Chen. Saup: Situation awareness uncertainty propagation on llm agent. arXiv preprint arXiv:2412.01033, 2024

    2024 arXiv

  35. [43]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 1 0 (2), 2023

    2023 arXiv

  36. [44]

    A survey of large language models in medicine: Progress, application, and challenge

    Hongjian Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam S Chen, Peilin Zhou, Junling Liu, et al. A survey of large language models in medicine: Progress, application, and challenge. arXiv preprint arXiv:2311.05112, 2023

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.