Pith. sign in

REVIEW 4 major objections 6 minor 25 references

Calibrated token log-probabilities, propagated through a Bayesian Network over agent workflow stages, give runtime uncertainty scores for LLM-based multi-agent actuarial systems while matching baseline predictive performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:13 UTC pith:XZ73UXCR

load-bearing objection The architecture and reproducibility are solid, but the central monitoring claim lacks empirical support—the uncertainty scores barely move between safe and perturbed runs. the 4 major comments →

arxiv 2607.25877 v1 pith:XZ73UXCR submitted 2026-07-28 cs.AI

Runtime Uncertainty Monitoring for LLM-Based Multi-Agent Systems Using Bayesian Networks

Bart Custers , Koorosh Aslansefat This is my paper
classification cs.AI
keywords multi-agent systemslarge language modelsuncertainty quantificationBayesian networkslog probabilitiesruntime monitoringactuarial risk modellingworkflow reliability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that uncertainty in a multi-agent LLM actuarial workflow can be monitored at runtime without sacrificing predictive accuracy. The authors convert length-normalised token log-probabilities from each agent into calibrated task-success probabilities and feed them into a Bayesian Network whose nodes represent workflow stages. The network then propagates these calibrated signals to produce an overall workflow confidence score, tracing how uncertainty accumulates across dependent agent outputs. A sympathetic reader would care because, if the approach works, high-stakes users could watch confidence drop in real time, identify the weakest workflow step, and decide when human review is needed—something per-agent accuracy alone cannot show.

Core claim

The central claim is that token-level log-probabilities—which are only weakly correlated with correctness—can be made useful for workflow-level uncertainty by first calibrating them into task-success probabilities and then propagating those probabilities through a Bayesian Network that models dependencies between agent stages. The paper demonstrates this on an actuarial risk-modelling workflow with four specialised agents coordinated by a central hub. Across three LLM backends, the system reproduces baseline RMSE while producing workflow-level uncertainty scores that visibly decrease as LLM temperature (and hence output randomness) increases. The network's final 'WorkflowOK' node, modelled a

What carries the argument

The Bayesian Network is the central object: nodes are workflow tasks (data preparation, modelling, review, explanation), directed edges are dependencies, and conditional probability tables quantify how each stage's success depends on its parents. Token log-probabilities enter through Eq. (5): a length-normalised log-probability summary is exponentiated to a confidence feature and then mapped by a calibration function f_cal, estimated on validation runs, to P(task success). This calibrated value becomes evidence for the corresponding node. The WorkflowOK node uses an AND-gate CPT so that the overall workflow is deemed successful only if all parent tasks succeed.

Load-bearing premise

The whole uncertainty score rests on the unstated calibration function f_cal that maps token log-probability summaries to task-success probabilities, and on hand-specified Bayesian Network conditional probability tables; if either misrepresents reality, the workflow-level uncertainty numbers are not meaningful.

What would settle it

Run the agent workflow many times with varied temperatures and perturbed inputs, record the Bayesian Network's posterior for WorkflowOK for each run, and compare it with an independent, ground-truth judgment of whether the workflow actually succeeded (e.g., final model accepted and validated against a held-out set). If the posterior does not track observed success rates—especially across backends or on unseen perturbations—or if the calibration function estimated on validation runs fails to transfer to new runs, the central claim collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Workflow-level uncertainty is substantially lower than individual agent-level confidence, showing that error propagation across stages is real and quantifiable (e.g., per-agent confidence near 80% can yield a 55% final WorkflowOK probability).
  • Raising LLM temperature decreases the Bayesian Network's posterior for workflow success, confirming that the monitor responds to controllable sources of randomness.
  • Backend choice matters: in the paper's tests, Qwen detected 90% of perturbed datasets while Llama 2 detected 45%, and workflow uncertainty scores varied widely across backends even when predictive performance was similar.
  • The Bayesian Network can flag which workflow node contributes most to overall uncertainty, offering a concrete target for intervention or human review.
  • The framework should be read as a runtime uncertainty-propagation monitor, not as a certificate of output correctness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same calibration-plus-Bayesian-Network pattern could transfer to other regulated multi-step LLM pipelines (e.g., medical triage or legal document review) wherever task-level success labels are available for calibration.
  • If the calibration function is reliably monotonic, workflow uncertainty could double as an anomaly detector: perturbed datasets that the agents fail to detect may still produce unusual posterior scores, as hinted by the paper's comparison of default and perturbed runs.
  • The hand-specified AND-gate and prior probabilities could be replaced by learned CPTs or a noisy-OR structure to better reflect real-world resilience, a direction the paper leaves open.
  • A direct validation step—comparing BN posteriors against actual, independently adjudicated workflow success—would test whether the uncertainty scores are meaningful, not just internally consistent.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a multi-agent system (MAS) for actuarial risk modelling in which four specialized LLM-based agents (data preparation, modelling, reviewing, explanation) are coordinated by a central hub. The main contribution is a runtime uncertainty-monitoring framework: token-level log-probabilities are length-normalized (Eq. 4), transformed into confidence features q_i, calibrated via an unspecified function f_cal (Eq. 5), and then entered as evidence into a hand-specified Bayesian Network (BN) whose CPTs are set a priori (e.g., the AND gate for WorkflowOK in Fig. 3). The BN propagates uncertainty to estimate P(WorkflowOK). The system is evaluated on three 7B LLM backends, with 10 default runs and 20 perturbed datasets. RMSE is comparable to a baseline, but the workflow-level uncertainty scores show little or no separation between default and perturbed conditions, and no end-to-end validation of the monitor's signal is provided.

Significance. If the proposed monitoring approach were validated, it would be a useful step toward interpretable, runtime uncertainty quantification for LLM-based MAS in regulated decision support. The paper's strengths include a clear architecture, public code and data, and a sensible motivation for calibrating token-level logprobs before using them as BN evidence. The RMSE reproduction of baseline actuarial performance is credible. However, the central claim—that the BN-based uncertainty signal can help identify unsafe workflow states—is not currently supported by the reported experiments: the uncertainty scores do not systematically differentiate default from perturbed runs, and no calibration or discrimination metrics are reported. The contribution is therefore at the proof-of-concept stage, and the paper's conclusions are overstated relative to the evidence.

major comments (4)
  1. [Table 2] The central monitoring claim is not supported by the reported metrics. Mean workflow uncertainty for default vs. perturbed runs: Llama 2 0.7345 vs. 0.7342 (Δ=0.0003), Llama 3.1 0.5506 vs. 0.5573 (Δ=0.0067), Qwen 0.6012 vs. 0.5833 (Δ=–0.0179). These differences are negligible and for Qwen run in the wrong direction, contradicting the claim that perturbed workflows exhibit higher uncertainty. With only 10 runs per condition, these are likely within noise. Please report per-run distributions, standard deviations, effect sizes, or a statistical test (e.g., Mann-Whitney) for default vs. perturbed, and an AUROC for detecting perturbed runs using P(WorkflowOK). Without this, the abstract's 'safety mechanism' claim is unsupported.
  2. [Eq. (5)] The calibration function f_cal is never specified. It is described only as 'estimated using validation runs', with no functional form (e.g., Platt scaling, isotonic regression, logistic), no training details, and no validation split. Since the calibrated values are the only evidence entered into the BN, the P(WorkflowOK) values in Tables 1–3 are not reproducible. This is a load-bearing omission: the paper's central mechanism depends on this function. Please provide its exact definition, fitting procedure, and the fitted parameters or a pointer to the code.
  3. [Sec. 4.2 / Fig. 3] The BN's CPTs are hand-set rather than learned from data. For example, WorkflowOK is an 'AND gate' with a prior of 0.99 (Fig. 3). Consequently, the scenario results in Table 1 (e.g., risky temperature reduces P(WorkflowOK) to 42.8%) are largely predetermined by the chosen CPT, not empirically derived from agent outputs. The paper should either learn the CPTs from logged agent success/failure data or provide a sensitivity analysis demonstrating that the qualitative conclusions in Table 1 are robust to reasonable variations in the prior and CPT. As it stands, these numbers are an illustration of the model, not a validation of the monitoring approach.
  4. [Sec. 5 / Table 3] The 20 perturbed datasets show no monotonic relationship between perturbation severity and uncertainty score. For instance, T13 (label noise low, target shift 0.2093) gives 0.6078 while T14 (label noise high, target shift 0.8303) gives 0.6065; T19 (counterfactual test 1, correlation shift 0.7575) gives 0.5727, lower than many less severe perturbations. No calibration curve, ECE, or AUROC is reported for P(WorkflowOK) against observed workflow success/failure. The authors should report P(WorkflowOK) for successful vs. failed runs and quantify discrimination (e.g., AUROC) and calibration. Without this, the framework cannot be considered a validated runtime uncertainty monitor.
minor comments (6)
  1. [Abstract] Typo: 'unsafe workflows states' should be 'unsafe workflow states'.
  2. [Fig. 1] The architecture figure is very low resolution; a vector graphic or higher-resolution version is needed.
  3. [Sec. 4.1] q_i = exp(s_i) is the geometric mean of the token probabilities; state this explicitly for clarity.
  4. [Table 2] No standard deviations, confidence intervals, or per-run statistics are reported; these are essential for comparing conditions with only 10 runs.
  5. [Appendix A.6] The denominator in the expression for p^p_k is written as 'nn' instead of n_p; also the notation convention differs from the rest of the appendix.
  6. [Tables 1–3] It is unclear whether the 'uncertainty score' is P(WorkflowOK) or 1−P(WorkflowOK). Table 1 reports P(WorkflowOK) as percentages (e.g., 87.1%), while Table 2 reports values like 0.7345. Please define the metric consistently.

Circularity Check

0 steps flagged

No significant circularity; workflow-level uncertainty is a model output from calibrated logprobs and hand-set CPTs, not an independently validated prediction.

full rationale

The derivation chain is explicit: token log-probabilities (Eqs. 1-4) are transformed by a calibration function f_cal (Eq. 5, fitted on validation runs), then used as node evidence in a Bayesian Network whose WorkflowOK node is defined by a hand-set AND-gate CPT (Fig. 3). Table 1 scenario probabilities follow from these definitions and assumptions, so they are illustrative sensitivity analyses rather than empirical predictions; describing them as 'insights' is a framing choice, not a circular reduction. No quantity is defined in terms of another predicted quantity, no fitted parameter is renamed as an independent prediction, and the one self-citation [3] (Donaldson et al., with Aslansefat as co-author) appears only as background for Bayesian-Network uncertainty propagation and is not load-bearing for the central claim. The paper itself acknowledges in Sec. 6 that the Bayesian Network is 'not as a proof of output correctness', which is an explicit validity limitation rather than hidden circularity. The absence of end-to-end validation of P(WorkflowOK) against observed workflow failure is a correctness/evidence gap, not a by-construction equivalence.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central uncertainty numbers are produced by a fitted calibration function and hand-set CPTs; the paper contributes the integration and the domain application, but the quantitative outputs rest on these free parameters and domain assumptions.

free parameters (3)
  • calibration function f_cal = not specified (estimated on validation runs)
    Eq. (5): P(X_i=success|q_i)=f_cal(q_i). The mapping from exp(mean logprob) to task success probability is fitted on validation data; its form (logistic, isotonic, etc.) and training details are not given, so it can absorb arbitrary scaling of logprobs and is a free parameter.
  • Bayesian Network conditional probability tables (CPTs) = e.g., P(WorkflowOK=1 | all parents true)=0.99; other tables not shown
    Sec. 4.2/Fig. 3: the CPTs (AND gate with prior 0.99) are chosen by hand to represent dependencies; they are not learned from data and directly determine the final workflow success probabilities.
  • Bayesian Network structure (DAG edges) = the graph in Fig. 2
    Sec. 4.2: the abstracted execution trace is assumed to be a DAG with given edges; structure is chosen by the authors, not inferred from data.
axioms (5)
  • domain assumption Log-probability summaries, after calibration, are valid estimates of task success probabilities
    Sec. 4.1 Eq. 5: the entire BN is fed with P(X_i=success|q_i)=f_cal(q_i); this is assumed without external validation that calibrated logprobs track actual success.
  • domain assumption The workflow dependencies are captured by the specified BN factorization
    Sec. 4.2 Eq. 6 and Fig. 2: the joint distribution factorizes per the graph; the paper acknowledges it abstracts away temporal ordering, feedback loops, and repeated revisions (Sec. 4.2, Sec. 6).
  • domain assumption The final accepted output of a stage is sufficient evidence of success
    Sec. 4.2: 'the final accepted output of that stage is used as the evidence for the corresponding node'; this discards information about revisions and failures during the stage.
  • domain assumption WorkflowOK succeeds only if all parent nodes are true (AND gate)
    Sec. 4.2 Fig. 3: the CPT is labelled an 'AND' gate; the choice of logical AND is a modeling assumption, not derived from data.
  • domain assumption f_cal estimated on validation runs transfers across temperatures and backends
    Table 1 varies LLM temperature for the scenario tests, but the calibration function is estimated on (presumably) default runs; the paper does not discuss whether f_cal remains valid at different temperatures.

pith-pipeline@v1.3.0-alltime-deepseek · 9324 in / 13431 out tokens · 110649 ms · 2026-08-01T01:13:16.710702+00:00 · methodology

0 comments
read the original abstract

This paper investigates how multi-agent systems (MAS)-based on large language models (LLMs) can support actuarial risk modelling, with a particular focus on uncertainty quantification. Actuarial workflows represent a high-stakes decision-support setting where unreliable outputs may lead to incorrect risk assessment, unfair pricing, and regulatory non-compliance. To address uncertainty introduced by the probabilistic nature of LLMs and dependencies between agents, a multi-agent framework is proposed in which specialised agents perform data preparation, modelling, review, and explanation tasks under a central hub. The main contribution is a novel approach to uncertainty propagation using token-level log-probabilities and a Bayesian Network. Importantly, log probabilities are not treated as direct probabilities of correctness or task success. Instead, length-normalised log-probability summaries are transformed into calibrated task-level confidence estimates before incorporation into the Bayesian Network. Results show that the framework reproduces baseline actuarial performance while providing additional insight into workflow stability and runtime uncertainty propagation.

Figures

Figures reproduced from arXiv: 2607.25877 by Bart Custers, Koorosh Aslansefat.

Figure 1
Figure 1. Figure 1: Sketch of the application’s architecture The framework supports three LLM backends, all relatively small in size, as the aim was to invoke backends that can be quickly run on local machines. The three models, Llama 2 7B, and Llama 3.1 8B [12], and Qwen2.5 7B [18], can all be characterized as decoder-only transformer models and are all available as open source model. 4 Uncertainty quantification 4.1 Log pro… view at source ↗
Figure 2
Figure 2. Figure 2: Bayesian Network representation of the actuarial workflow, including posteriors for each phase success (1) or failure (0). The Bayesian Network used in this study represents an abstracted execution trace of the workflow rather than the full control logic of the agentic system. We do not model temporal ordering beyond direct parent-child dependencies, nor do we explicitly represent task priority, repeated e… view at source ↗
Figure 3
Figure 3. Figure 3: CPT example of WorkflowOK, depending on three parent nodes. The “AND” gate label indicates that WorkflowOK succeeds only if all parent nodes pass. Along the workflow run, new evidence is included in the network, and pos￾terior probabilities are updated. The resulting network provides a dependency graph of uncertainty propagation within the workflow, as visualized in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Uncertainty distribution, compared between original (default) and test datasets, for workflow runs with Qwen backend. The four individual points represent the uncer￾tainty outcomes from the test scenarios in [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    arXiv 2512.03816 (12 2025)

    Chauvin, T., Merrer, E.L., Taïani, F., Tredan, G.: Log probability tracking of llm apis. arXiv 2512.03816 (12 2025). https://doi.org/10.48550/arXiv.2512.03816

    doi:10.48550/arxiv.2512.03816 2025

  2. [2]

    arXiv 2308.10848 (10 2023)

    Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Chan, C.M., Yu, H., Lu, Y., Hung, Y.H., Qian, C., Qin, Y., Cong, X., Xie, R., Liu, Z., Sun, M., Zhou, J.: Agent- verse: Facilitating multi-agent collaboration and exploring emergent behaviors. arXiv 2308.10848 (10 2023). https://doi.org/10.48550/arXiv.2308.10848

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.10848 2023

  3. [3]

    arXiv preprint arXiv:2607.00972 (2026)

    Donaldson, L., Walker, C., Aslansefat, K., Papadopoulos, Y.: Bayesian uncertainty propagation for agentic rag pipelines: A proof-of-concept study on multi-hop ques- tion answering. arXiv preprint arXiv:2607.00972 (2026)

    Pith/arXiv arXiv 2026

  4. [4]

    arXiv 2506.17419 (6 2025)

    Duan, J., Diffenderfer, J., Madireddy, S., Chen, T., Kailkhura, B., Xu, K.: Uprop: Investigating the uncertainty propagation of llms in multi-step agentic decision- making. arXiv 2506.17419 (6 2025). https://doi.org/10.48550/arXiv.2506.17419

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.17419 2025

  5. [5]

    arXiv 2506.18942 (6 2025)

    Hatzesberger, S., Nonneman, I.: Advanced applications of generative ai in actuarial science: Case studies beyond chatgpt. arXiv 2506.18942 (6 2025). https://doi.org/10.48550/arXiv.2506.18942

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.18942 2025

  6. [6]

    arXiv 2505.24201 (2025)

    He, X., Wu, D., Zhai, Y., Sun, K.: Sentinelagent: Graph-based anomaly detection in multi-agent systems. arXiv 2505.24201 (2025). https://doi.org/10.48550/arXiv.2505.24201 12 B. Custers et al

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.24201 2025

  7. [7]

    A Bayesian Approach to Harnessing the Power of LLMs in Authorship Attribution

    Hu, Z., Zheng, T., Huang, H.: A bayesian approach to harnessing the power of llms in authorship attribution. arXiv 2410.21716 (10 2024). https://doi.org/10.48550/arXiv.2410.21716

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2410.21716 2024

  8. [8]

    https://actuaries.org/paper/artificial-intelligence- governance-framework/ [Accessed 3/12/25] (2024)

    International Actuarial Association: Artificial intelligence governance framework - general actuarial practice. https://actuaries.org/paper/artificial-intelligence- governance-framework/ [Accessed 3/12/25] (2024)

    2024

  9. [9]

    arXiv 1711.11279 (6 2017)

    Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., Sayres, R.: Interpretability beyond feature attribution: Quantitative test- ing with concept activation vectors (tcav). arXiv 1711.11279 (6 2017). https://doi.org/10.48550/arXiv.1711.11279

    Pith/arXiv arXiv doi:10.48550/arxiv.1711.11279 2017

  10. [10]

    Expert Systems with Applications42(4), 1917–1926 (2015)

    Liu, Z., Liu, Y., Cai, B., Zheng, C.: An approach for developing diagnostic bayesian network based on operation procedures. Expert Systems with Applications42(4), 1917–1926 (2015). https://doi.org/https://doi.org/10.1016/j.eswa.2014.10.020, https://www.sciencedirect.com/science/article/pii/S0957417414006447

    work page doi:10.1016/j.eswa.2014.10.020 1917

  11. [11]

    arXiv 2002.07650 (2 2020)

    Malinin, A., Gales, M.: Uncertainty estimation in autoregressive structured pre- diction. arXiv 2002.07650 (2 2020). https://doi.org/10.48550/arXiv.2002.07650

    Pith/arXiv arXiv doi:10.48550/arxiv.2002.07650 2002

  12. [12]

    https://ai.meta.com/blog/meta-llama-3-1/ [Ac- cessed 25/1/2026] (7 2024)

    Meta AI: Introducing llama 3.1. https://ai.meta.com/blog/meta-llama-3-1/ [Ac- cessed 25/1/2026] (7 2024)

    2026

  13. [13]

    In: Proceedings of the 31st ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining V.2

    Mohammadi, M., Li, Y., Lo, J., Yip, W.: Evaluation and benchmarking of llm agents: A survey. In: Proceedings of the 31st ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining V.2. pp. 6129–6139. Association for Computing Machinery (ACM) (8 2025). https://doi.org/10.1145/3711896.3736570

    arXiv 2025

  14. [14]

    arXiv 2505.15918 (8 2025)

    Nafar, A., Venable, K.B., Cui, Z., Kordjamshidi, P.: Extracting probabilistic knowl- edge from large language models for bayesian network parameterization. arXiv 2505.15918 (8 2025). https://doi.org/10.48550/arXiv.2505.15918

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.15918 2025

  15. [15]

    Applied Soft Computing52, 109–119 (3 2017)

    Oreski, D., Oreski, S., Klicek, B.: Effects of dataset characteristics on the per- formance of feature selection techniques. Applied Soft Computing52, 109–119 (3 2017). https://doi.org/10.1016/j.asoc.2016.12.023

    work page doi:10.1016/j.asoc.2016.12.023 2017

  16. [16]

    Risks10, 230 (12 2022)

    Owens, E., Sheehan, B., Mullins, M., Cunneen, M., Ressel, J., Castignani, G.: Explainable artificial intelligence (xai) in insurance. Risks10, 230 (12 2022). https://doi.org/10.3390/risks10120230

    work page doi:10.3390/risks10120230 2022

  17. [17]

    Morgan Kaufmann Pub- lishers Inc., San Francisco, USA (1988)

    Pearl, J.: Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann Pub- lishers Inc., San Francisco, USA (1988). https://doi.org/10.1016/C2009-0-27609-4

    doi:10.1016/c2009-0-27609-4 1988

  18. [18]

    https://huggingface.co/Qwen/Qwen2.5-7B-Instruct [Accessed 25/1/2026] (2025)

    Qwen Team: Qwen2.5-7b-instruct (model card). https://huggingface.co/Qwen/Qwen2.5-7B-Instruct [Accessed 25/1/2026] (2025)

    2026

  19. [19]

    agentic ai: A con- ceptual taxonomy, applications and challenges

    Sapkota, R., Roumeliotis, K.I., Karkee, M.: Ai agents vs. agentic ai: A con- ceptual taxonomy, applications and challenges. arXiv 2505.10468 (5 2025). https://doi.org/10.48550/arXiv.2505.10468

    doi:10.48550/arxiv.2505.10468 2025

  20. [20]

    arXiv 2501.06322 (1 2025)

    Tran, K.T., Dao, D., Nguyen, M.D., Pham, Q.V., O’Sullivan, B., Nguyen, H.D.: Multi-agent collaboration mechanisms: A survey of llms. arXiv 2501.06322 (1 2025). https://doi.org/10.48550/arXiv.2501.06322

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.06322 2025

  21. [21]

    Frontiers of Computer Science18(12 2024)

    Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., Zhao, W.X., Wei, Z., Wen, J.: A survey on large language model based autonomous agents. Frontiers of Computer Science18(12 2024). https://doi.org/10.1007/s11704-024-40231-1

    doi:10.1007/s11704-024-40231-1 2024

  22. [22]

    arXiv 2308.08155 (10 2023)

    Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A.H., White, R.W., Burger, D., Wang, C.: Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv 2308.08155 (10 2023). https://doi.org/10.48550/arXiv.2308.08155 Uncertainty Monitoring for LLM Agents 13

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.08155 2023

  23. [23]

    In: Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering

    Xu, M., Gan, Q., Zhu, Z., Qin, H.: Logprobs know uncertainty: Fighting llm hallu- cinations. In: Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering. pp. 1242–1243. Association for Computing Machinery (7 2025). https://doi.org/10.1145/3696630.3731433

    arXiv 2025

  24. [24]

    arXiv 2411.14033 (12 2024)

    Yang, Y., Peng, Q., Wang, J., Wen, Y., Zhang, W.: Llm-based multi-agent systems: Techniques and business perspectives. arXiv 2411.14033 (12 2024). https://doi.org/10.48550/arXiv.2411.14033

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.14033 2024

  25. [25]

    arXiv 2503.16416 (3 2025)

    Yehudai, A., Eden, L., Li, A., Uziel, G., Zhao, Y., Bar-Haim, R., Cohan, A., Shmueli-Scheuer, M.: Survey on evaluation of llm-based agents. arXiv 2503.16416 (3 2025). https://doi.org/10.48550/arXiv.2503.16416 A Appendix: Dataset severity indicators To quantify how strongly a dataset has changed between a baseline datasetDb and a perturbed datasetDp, sever...

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.16416 2025