Trustworthy Agents for Electronic Health Records through Confidence Estimation

Minbyul Jeong; Mujeen Sung; Yongwoo Song

REVIEW 3 major objections 5 minor 32 references

Trustworthy Agents for Electronic Health Records through Confidence Estimation

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An EHR question-answering agent that estimates confidence step-by-step can stay accurate where reliability is strictly required, while baseline agents that answer everything collapse to zero.

desk verdict The headline gains are partly an artifact of test-set-optimal threshold selection, but the step-wise confidence mechanism and the empirical comparison are worth serious attention. read the letter →

arxiv 2508.19096 v1 pith:RJLWBRIR submitted 2025-08-26 cs.AI

Yongwoo Song , Minbyul Jeong , Mujeen Sung This is my paper

classification cs.AI

keywords confidenceestimationelectronichealthrecordshallucinationcontrolselectiveanswerabstentionLLMagenttext-to-SQLreliabilitymetricclinicaldecisionsupport

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to judge medical database assistants by both accuracy and willingness to abstain: Hallucination-Controlled Accuracy at k% (HCAcc@k%) reports the best accuracy an agent can achieve while keeping the share of wrong answers among provided answers at or below (100-k)%. On MIMIC-III and eICU EHR benchmarks, the authors' TrustEHRAgent reaches 44.23% and 25.34% accuracy at HCAcc@70%, where confidence-augmented baseline agents score 0.0% because every answer they give above the confidence threshold is wrong. The agent estimates confidence at each reasoning step and again at the final answer, using token log-probabilities to form a weighted confidence score. The central claim is that step-wise confidence estimation, not raw reasoning ability, is what lets a clinical agent remain useful under strict patient-safety constraints.

What carries the argument

HCAcc@k% = the maximum over thresholds τ of Overall Accuracy subject to Hallucination Rate ≤ (100-k)/100, where Hallucination Rate is the fraction of wrong answers among provided answers and Overall Accuracy is correct answers divided by all questions. The carrying mechanism is the stepwise confidence estimator: the agent verbalizes a 0-10 confidence at each reasoning step, then a separate LLM-based estimator takes query, reasoning history, and answer, computes P(S=i)=exp(logprob_i) over five confidence levels, and returns the weighted average normalized by Smax. That continuous C gives the agent an abstention rule C ≥ τ, and the metric rewards methods that can set τ to filter errors.

What would settle it

Run the same experiments with the confidence threshold fixed on a validation split, never touched on the test set, then compute HCAcc@70% on MIMIC-III and eICU; if TrustEHRAgent's margin over baseline agents drops substantially, the central claim fails. A second check: recompute the confidence score with true softmax normalization over the five log-probabilities instead of raw exp(logprob) sums; if the ranking of answers changes, the reported thresholds are not interpretable as stated.

Watch

Extended reading notes

Core claim

TrustEHRAgent couples a code-based EHR reasoning loop with stepwise confidence ratings and a final confidence estimator, producing a score C in [0,1] for each candidate answer. During execution, the agent writes a 0-10 confidence comment at each code step; after the final answer, a separate confidence-estimator call, conditioned on the question, the enriched reasoning history, and the answer, maps five discrete confidence levels into a continuous score via a log-probability weighted average. Answers are emitted only if C passes threshold τ. With τ chosen to satisfy a hallucination-rate cap, the agent achieves HCAcc@70% of 44.23% on MIMIC-III and 25.34% on eICU, while the same confidence pipe

Load-bearing premise

The reported HCAcc@k% numbers depend on two assumptions: that a confidence threshold chosen by scanning the test set will behave the same in deployment, and that the confidence score from Eq. 2 is a properly normalized probability; if either fails, the large gaps over baselines at k=70% may shrink or disappear.

Editorial extensions

If this is right

Clinical EHR assistants can be deployed under an explicit safety budget: choose a risk tolerance, set k, and the agent answers only when its confidence clears the corresponding bar, otherwise it abstains.
Accuracy-only comparisons can mislead: two agents with close scores at HCAcc@0% behave very differently under strict reliability constraints, so high-stakes evaluations should report HCAcc curves.
Stepwise confidence signals improve the final confidence estimate; removing them degrades performance specifically at high reliability thresholds.
At HCAcc@90% even TrustEHRAgent drops to 3.27% (MIMIC-III) and 2.24% (eICU), so operating at extreme reliability remains unsolved.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The HCAcc@k% values presume the threshold τ is selected on the same test set used for reporting; a deployment protocol that fixes τ on a validation split would likely produce lower numbers, and the paper does not report that protocol.
Because HCAcc@k% does not depend on EHR content, it transfers to any selective question-answering setting where false answers are costly, such as legal or financial document querying.
The stepwise log-probability estimator could be combined with better calibration techniques (temperature scaling, ensembles) to push the HCAcc@90% numbers up, a direction the paper names as future work.
A direct replication on an open-weight model with accessible token probabilities would test whether the gap over baselines persists when the backbone is not the proprietary model used here.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The main thing you should know: the 44.23%/25.34% headline gains at HCAcc@70% are computed by taking the maximum over the confidence threshold on the test set itself. That is an oracle evaluation. With only 580 questions, the in-sample maximum can easily find a threshold that happens to admit mostly correct answers, especially at high k% where the accepted set is small. So the magnitude of the claimed advantage over the 0.0% baselines is not yet established under a realistic protocol. This is the paper's biggest soft spot.

Where credit is due: the paper is honest about its limitations, the full HCAcc@k% curves are in the figures, and the ablation in Figure 4 shows that step-wise confidence estimation helps at high thresholds. The integration of verbalized step-wise confidence with token-log-probability weighting is a reasonable design, and the experiments include confidence-augmented baselines, which is more than many papers do.

The metric itself is not new. HCAcc@k% is a re-parameterization of the standard accuracy-coverage trade-off with a precision constraint from the selective prediction literature (Chow, El-Yaniv and Wiener, etc.), and that literature is not cited. Eq. 2 is also sloppy: P(S=i) = exp(logprob_i) is not normalized, so the "weighted average" is not a proper expectation over a probability distribution. That's a minor issue for monotonicity, but it should be fixed.

No error bars are reported, and the test set is a single split of 580 questions. A validation-based threshold, or a fixed a priori threshold, plus some variance estimate, would make the empirical claims much stronger.

Bottom line: the core research question is important—how to make EHR agents abstain when uncertain—and the step-wise confidence approach is plausible and worth testing. But the headline numbers are likely to shrink when the evaluation protocol is cleaned up. This paper deserves a serious referee, but with a clear request to recompute the numbers under a non-oracle threshold selection, fix the normalization, and report variance. If those changes are made, I could see this being a useful reference for the clinical LLM reliability community.

Referee Report

3 major / 5 minor

Summary. The paper proposes Hallucination-Controlled Accuracy at k% (HCAcc@k%), a metric that measures the maximum overall accuracy achievable while keeping the hallucination rate below a specified threshold, and introduces TrustEHRAgent, a confidence-aware EHR question-answering agent with step-wise confidence estimation. On the MIMIC-III and eICU QA datasets, the agent is reported to achieve HCAcc@70% of 44.23% and 25.34%, respectively, while confidence-augmented SQLAgent and PythonAgent baselines achieve 0%, and it retains near-baseline accuracy at HCAcc@0% (62.13% and 59.82%). The paper argues that traditional accuracy metrics hide reliability differences that matter for clinical safety.

Significance. If the reported results hold, the paper makes two useful contributions: a reliability-aware evaluation metric for selective answering systems and an agent design that integrates step-wise confidence signals. The full HCAcc curves in Figure 3 and the availability of code are strengths, and the comparison against multiple baselines is informative. However, the central quantitative claims are not yet established because HCAcc@k% is maximized over the confidence threshold on the same test set used for reporting, and the confidence score in Eq. (2) is not a properly normalized probability. These issues directly affect the headline 44.23p/25.34p gaps, so the current evidence supports the approach as promising but not yet as a validated deployable advantage.

major comments (3)

[Sec. 3.4, Eq. (5); Sec. 5.1, Table 1] HCAcc@k% is defined as max_τ OA(τ) subject to HR(τ) ≤ (100−k)/100, and the reported values are obtained by optimizing τ on the same 580-question test sets used for reporting. This is an oracle evaluation: the operating point is chosen after seeing test labels. Especially at k=70–90%, the accepted set Q_τ is small, so the maximum over τ can be substantially higher than the accuracy at any fixed or validation-selected threshold. The headline claim that TrustEHRAgent achieves 44.23%/25.34% while baselines achieve 0.0% therefore needs to be re-evaluated with τ selected on a validation split (or via nested cross-validation), with the chosen τ values reported. Bootstrap confidence intervals should also be provided, since n=580 per dataset makes the high-k values unstable.
[Sec. 3.2, Eq. (2)] The confidence score is defined as C = (Σ i·P(S=i))/Smax with P(S=i)=exp(logprob_i). This is not a normalized probability distribution: summing exp(logprob_i) over the five confidence tokens does not sum to 1, and the resulting C is not guaranteed to lie in [0,1]. The missing probability mass is ignored, so the score is not a well-defined expected level. This matters because C is the quantity compared against τ; an unnormalized, token-set-dependent weighting can change the ordering of examples and hence the reported HCAcc@k% curves. Please define P(S=i) via a softmax over the five levels (or explicitly normalize by the sum), and clarify whether logprob_i are log-probabilities or logits.
[Sec. 5.1, Figure 3] No uncertainty quantification is provided for any HCAcc@k% value. The central comparison is between 44.23% and 0.0% at a single operating point, but at k=70% the baselines' 0.0% could arise from a single unfavorable threshold while the proposed method's value is the best threshold on the test set. Bootstrap resampling over questions or multiple evaluation splits is needed to establish that the gap is stable and not an artifact of in-sample threshold selection. This is particularly important because the paper's own Limitations section notes the need for better calibration methods.

minor comments (5)

[Table 1] The rows without confidence estimation appear to list only two numbers; please specify which HCAcc@k% columns are reported for each baseline. Currently the layout is ambiguous.
[Figure 2] The query text 'sc 20 fe ng intake' appears garbled. If this is from the actual EHR data, please add a gloss or footnote; otherwise this looks like a rendering error.
[Abstract and Conclusion] The phrase 'improvements of 44.23%p' is imprecise: Table 1 reports an absolute HCAcc@70% value of 44.23% for TrustEHRAgent and 0% for baselines. The percentage-point improvement is 44.23p, not a relative improvement. Please clarify.
[Appendix D] The confidence-aware baselines are said to use 'the same step-wise confidence prompting and final confidence estimator.' It would be useful to state explicitly whether SQLAgent and PythonAgent actually received the step-wise confidence rating instruction during reasoning, or only the final estimator. This affects whether the comparison isolates the agent architecture or the confidence mechanism.
[Sec. 3.2] The text says 'N (= 5) discrete levels (S ∈ {0, 1, ..., Smax})' and later sets Smax=4. This is consistent, but the notation is confusing; consider writing S ∈ {0,1,2,3,4} explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HCAcc@k% is an explicitly oracle-threshold evaluation metric and the reported gains are empirical, not forced by the metric's definition.

full rationale

HCAcc@k% (Eq. 5) is explicitly defined as the maximum over τ of OA(τ) subject to HR(τ) ≤ (100−k)/100. This is an oracle definition of an evaluation metric, not a hidden fitted parameter relabeled as a prediction; every method is scored by the same envelope. The headline numbers in Table 1 (44.23%, 25.34%) are computed from actual correctness labels and the model's confidence outputs, so nothing in the definition forces TrustEHRAgent to beat SQLAgent/PythonAgent—indeed the baselines score 0.0, which is an empirical result. The paper does not rely on any load-bearing self-citation: it builds on external prior work (Shi et al. 2024; Roucher et al. 2025) and applies the same confidence-estimation pipeline to the confidence-aware baselines. The in-sample maximization over τ is a genuine evaluation-protocol limitation—the reported operating point is selected on the same test set used for reporting and may not reproduce under a fixed a priori threshold—but that is a matter of external validity or overfitting, which is a correctness/reproducibility concern rather than circularity. The step-wise confidence estimates are prompt-based signals, not mathematical assumptions that entail the result. The Limitations section also honestly notes that calibration at high thresholds remains imperfect. Accordingly, no circular step can be exhibited from the paper's own equations or citation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are postulated. HCAcc@k% is a defined evaluation metric, not an invented entity of the kind this ledger targets.

free parameters (3)

confidence threshold τ = not reported; selected as argmax in Eq. 5 on the evaluation set
The reported HCAcc@k% is the maximum over τ, so the threshold is a free parameter tuned to test data.
number of confidence levels N (Smax=4) = 5
Choice of discrete levels in the confidence estimator; affects the resolution of C.
step-wise rating scale 0-10 = 10 levels
Prompt instruction for verbalized confidence; a design choice.

assumptions (4)

ad hoc to paper Token log-probabilities over the 5 confidence tokens can be combined as exp(logprob_i) to form a scalar confidence C in [0,1]
Eq. 2 introduces this combination without normalization or derivation; as written it is not a proper probability distribution.
domain assumption The 580 QA pairs per dataset from EHRSQL/EHRAgent are representative of real clinical queries
Stated in Limitations: the datasets may not fully represent real-world clinical settings (Sec. Limitations).
domain assumption Exact match between predicted and reference answer is a valid correctness criterion
Sec. 4.3 defines correctness by exact match, which penalizes equivalent but differently formatted answers.
domain assumption Step-wise verbalized confidence ratings carry signal about answer correctness
Sec. 3.2 uses these ratings as inputs; Sec. 6.1 provides an ablation, so it is partially tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Trustworthy Agents for Electronic Health Records through Confidence Estimation." pith.science (2026). https://pith.science/paper/RJLWBRIR

@misc{pith2026250819096,
  author       = {Pith},
  title        = {Pith review of: Trustworthy Agents for Electronic Health Records through Confidence Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RJLWBRIR}},
  note         = {Machine review of arXiv:2508.19096}
}

read the original abstract

Large language models (LLMs) show promise for extracting information from Electronic Health Records (EHR) and supporting clinical decisions. However, deployment in clinical settings faces challenges due to hallucination risks. We propose Hallucination Controlled Accuracy at k% (HCAcc@k%), a novel metric quantifying the accuracy-reliability trade-off at varying confidence thresholds. We introduce TrustEHRAgent, a confidence-aware agent incorporating stepwise confidence estimation for clinical question answering. Experiments on MIMIC-III and eICU datasets show TrustEHRAgent outperforms baselines under strict reliability constraints, achieving improvements of 44.23%p and 25.34%p at HCAcc@70% while baseline methods fail at these thresholds. These results highlight limitations of traditional accuracy metrics in evaluating healthcare AI agents. Our work contributes to developing trustworthy clinical agents that deliver accurate information or transparently express uncertainty when confidence is low.

Figures

Figures reproduced from arXiv: 2508.19096 by the authors.

**Figure 1.** Performance comparison under reliability constraints using the HCAcc@k% metric, where k% represents the minimum required reliability. Traditional accuracy metrics (A) cannot identify the Patient Safety Risk Area, a region where strict reliability is essential for patient safety. In contrast, while (B) PythonAgent completely fails at k ≥ 60%, (C) Our TrustEHRAgent maintains stable performance under high reliability… view at source ↗

**Figure 2.** Overview of TrustEHRAgent framework. The left panel shows step-wise confidence estimation during [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Performance-reliability trade-off analy [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Comparison of confidence estimation approaches: (A) Our weighted sum approach based on log probabilities, (B) Binary classification, and (C) Discrete scoring. for clinical AI safety. The consistently high confidence validates the reliability of complex analytical pr…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 15 canonical work pages

[1]

Vibhor Agarwal, Yiqiao Jin, Mohit Chandra, Munmun De Choudhury, Srijan Kumar, and Nishanth Sastry. 2024. Medhalu: Hallucinations in responses to healthcare queries by large language models. arXiv preprint arXiv:2409.19492

arXiv 2024
[2]

Muhammad Aurangzeb Ahmad, Ilker Yaramis, and Taposh Dutta Roy. 2023. Creating trustworthy llms: Dealing with hallucinations in healthcare ai. arXiv preprint arXiv:2311.01463

arXiv 2023
[3]

Hiba Ahsan, Denis Jered McInerney, Jisoo Kim, Christopher Potter, Geoffrey Young, Silvio Amir, and Byron C Wallace. 2024. Retrieving evidence from ehrs with llms: possibilities and challenges. Proceedings of machine learning research, 248:489

work page 2024
[4]

Jiuhai Chen and Jonas Mueller. 2023. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. arXiv (Cornell University)

work page 2023
[5]

Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, and Roi Reichart. 2025. Inside-out: Hidden factual knowledge in llms. arXiv preprint arXiv:2503.15299

arXiv 2025
[6]

Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, and 1 others. 2024. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature medicine, 30(9):2613--2622

work page 2024
[7]

Jiuzhou Han, Wray Buntine, and Ehsan Shareghi. 2024. Towards uncertainty-aware language agent. arXiv preprint arXiv:2401.14016

arXiv 2024
[8]

Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2020. How can we know when language models know? on the calibration of language models for question answering. arXiv (Cornell University)

work page 2020

Show all 32 references

[9]

Gyubok Lee, Hyeonji Hwang, Seongsu Bae, Yeonsu Kwon, Woncheol Shin, Seongjun Yang, Minjoon Seo, Jong-Yeup Kim, and Edward Choi. 2022. Ehrsql: A practical text-to-sql benchmark for electronic health records. Advances in Neural Information Processing Systems, 35:15589--15601

2022
[10]

Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Halueval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449--6464

2023
[11]

Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL https://arxiv. org/abs/2109.07958

2021 arXiv
[12]

Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187

2023 arXiv
[13]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634

2023 arXiv
[14]

Mat \'e o Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Mueller, and Llu \' s M \`a rquez. 2024. Factual confidence of llms: on reliability and robustness of current estimators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin...

2024
[15]

Gaya Mehenni and Amal Zouaq. 2025. Medhal: An evaluation dataset for medical hallucination detection. arXiv preprint arXiv:2504.08596

2025
[16]

Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. 2024. Llms know more than they show: On the intrinsic representation of llm hallucinations. arXiv preprint arXiv:2410.02707

2024 arXiv
[17]

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2023. Med-halt: Medical domain hallucination test for large language models. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 314--334

2023
[18]

Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. Advances in Neural Information Processing Systems, 36:36339--36348

2023
[19]

Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. 2024. Llm-based agentic systems in medicine and healthcare. Nature Machine Intelligence, 6(12):1418--1420

2024
[20]

Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. 2025. `smolagents`: a smol library to build great agentic systems. https://github.com/huggingface/smolagents

2025
[21]

Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce Ho, Carl Yang, and May Dongmei Wang. 2024. Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records. In Proceedings of the 2024 Conference...

2024
[22]

Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv. org/abs/2303.11366

2023 arXiv
[23]

Ping Wang, Tian Shi, and Chandan K Reddy. 2020. Text-to-sql generation for question answering on electronic medical records. In Proceedings of The Web Conference 2020, pages 350--361

2020
[24]

Weixuan Wang, Barry Haddow, Alexandra Birch, and Wei Peng. 2023. Assessing the reliability of large language model knowledge. arXiv preprint arXiv:2310.09820

2023 arXiv
[25]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[26]

Chengyan Wu, Zehong Lin, Wenlong Fang, and Yuyan Huang. 2023. A medical diagnostic assistant based on llm. In China Health Information Processing Conference, pages 135--147. Springer

2023
[27]

Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063

2023 arXiv
[28]

Daniel Yang, Yao-Hung Hubert Tsai, and Makoto Yamada. 2024. https://arxiv.org/abs/2412.14737 On verbalized confidence scores for llms . Preprint, arXiv:2412.14737

2024 arXiv
[29]

Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B Costa, Mona G Flores, and 1 others. 2022. A large language model for electronic health records. NPJ digital medicine, 5(1):194

2022
[30]

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

2023
[31]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[32]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Vibhor Agarwal, Yiqiao Jin, Mohit Chandra, Munmun De Choudhury, Srijan Kumar, and Nishanth Sastry. 2024. Medhalu: Hallucinations in responses to healthcare queries by large language models. arXiv preprint arXiv:2409.19492

arXiv 2024

[2] [2]

Muhammad Aurangzeb Ahmad, Ilker Yaramis, and Taposh Dutta Roy. 2023. Creating trustworthy llms: Dealing with hallucinations in healthcare ai. arXiv preprint arXiv:2311.01463

arXiv 2023

[3] [3]

Hiba Ahsan, Denis Jered McInerney, Jisoo Kim, Christopher Potter, Geoffrey Young, Silvio Amir, and Byron C Wallace. 2024. Retrieving evidence from ehrs with llms: possibilities and challenges. Proceedings of machine learning research, 248:489

work page 2024

[4] [4]

Jiuhai Chen and Jonas Mueller. 2023. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. arXiv (Cornell University)

work page 2023

[5] [5]

Zorik Gekhman, Eyal Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, and Roi Reichart. 2025. Inside-out: Hidden factual knowledge in llms. arXiv preprint arXiv:2503.15299

arXiv 2025

[6] [6]

Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, and 1 others. 2024. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature medicine, 30(9):2613--2622

work page 2024

[7] [7]

Jiuzhou Han, Wray Buntine, and Ehsan Shareghi. 2024. Towards uncertainty-aware language agent. arXiv preprint arXiv:2401.14016

arXiv 2024

[8] [8]

Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2020. How can we know when language models know? on the calibration of language models for question answering. arXiv (Cornell University)

work page 2020