Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model

Dan Turner; Gili Focht; Hadas Ben-Atya; Moti Freiman; Naama Gavrielov; Ruth Cytter-Kuint; Talar Hagopian; Zvi Badash

REVIEW 4 major objections 6 minor 36 references

Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An agent that pools six prompt answers and issues verdicts as Definitely, Likely, or Uncertain outperforms entropy-based uncertainty aggregation for automated Hebrew radiology report labeling.

desk verdict Useful empirical study in a low-resource language, but the headline agent-vs-entropy comparison is confounded by model scale (70B agent vs 8B baselines), and one reported F1 number is internally inconsistent. read the letter →

arxiv 2502.01691 v1 pith:HI4BJEVC submitted 2025-02-02 cs.CL cs.AI

Hadas Ben-Atya , Naama Gavrielov , Zvi Badash , Gili Focht , Ruth Cytter-Kuint , Talar Hagopian , Dan Turner , Moti Freiman This is my paper · ORCID

classification cs.CLcs.AI

keywords DeepLearningLLMUncertaintyEstimationRadiologyReportagent-baseddecisionmodelBayesianpromptensemblesHebrewstructureddataextraction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automated extraction of structured findings from free-text radiology reports is unreliable, especially in low-resource clinical languages like Hebrew, where models can be confidently wrong. This paper tries to establish that an agent-based decision model—an LLM that reviews six semantically equivalent prompt answers and their explanations and issues a verdict in one of five confidence categories—yields both better labels and better-calibrated uncertainty than entropy-based pooling of the same prompts. On 462 manually annotated reports spanning 23 organ–finding combinations, the agent reaches F1 0.3967, recall 0.6437, and Cohen's Kappa 0.3006; discarding cases it flags with uncertainty ≥0.5 raises F1 to 0.4787 and Kappa to 0.4258. The broader contribution is a path to trustworthy open-source LLM use in high-stakes medical text processing, with uncertainty histograms that separate correct from incorrect predictions.

What carries the argument

The central mechanism is the Agent Decision Model: an LLM (Llama 3 70B) that ingests the six answers and explanations from the BayesPE prompt ensemble and issues a structured JSON verdict in five confidence levels, with uncertainty assigned by hand as 0 for Definitely, 0.5 for Likely, and 1 for Uncertain. The agent is designed to weigh response consistency, explanation clarity, and signs of ambiguity. This categorical self-assessment, rather than a weighted entropy formula, is what the paper credits for the calibration separation and the post-filtering gains.

What would settle it

Compute the actual error rate among predictions the agent labels Definitely, Likely, and Uncertain on the 462-report test set. If 'Definitely' predictions are not substantially more accurate than 'Likely' ones, or if discarding all cases with uncertainty ≥0.5 does not remove a disproportionate share of errors, the claimed calibration advantage is falsified.

Watch

Extended reading notes

Core claim

The authors report that uncertainty for LLM-based structured data extraction is better modeled by an LLM agent's categorical self-assessment than by entropy computed over prompt ensembles. The agent receives the six yes/no answers and free-text explanations produced by Llama 3.1 8B under six semantically equivalent prompts (BayesPE), and using Llama 3 70B as the agent, it synthesizes them into one of five decisions: Definitely Yes, Likely Yes, Definitely No, Likely No, or Uncertain. These map to uncertainty values 0, 0.5, and 1. The agent's average median uncertainty is 0 for correct predictions and about 0.5 for incorrect predictions, a wider separation than uniform, linearly optimized, or MLP-learned weights. The resulting F1 of 0.3967 (recall 0.6437, Kappa 0.3006) exceeds all entropy baselines, and excluding high-uncertainty cases raises F1 to 0.4787 and Kappa to 0.4258.

Load-bearing premise

The paper's calibration benefit rests on the assumption that the agent's five confidence categories track true error probability monotonically, with uncertainty values 0, 0.5, and 1 hand-assigned; no calibration curve or per-category error rates are reported.

Editorial extensions

If this is right

Filtering out cases the agent labels Uncertain or Likely (uncertainty ≥ 0.5) improves F1 from 0.3967 to 0.4787 and Kappa from 0.3006 to 0.4258, so selectively referring high-uncertainty cases to human readers is a viable workflow.
Open-source Llama 3.1 can extract structured findings from Hebrew radiology reports more reliably when its six prompt answers are pooled by an agent than by entropy weighting, which matters for privacy-sensitive clinical data.
The agent's uncertainty histograms separate correct from incorrect predictions better than the entropy baselines, meaning the confidence labels carry practical information about when to trust the model.
The method's recall advantage (0.6437 versus 0.3977 for the MLP baseline) means it finds more true positives, at the cost of lower precision and accuracy, so task-specific thresholds may be needed.
Because the prompts are semantically equivalent and the agent output is structured JSON, the same pipeline can be adapted to other organs, findings, and languages without retraining the backbone LLM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the paper fixes the agent's uncertainty values at 0, 0.5, and 1 by hand; a natural next test is to estimate the empirical error rate within each of the five confidence categories and replace the hand-set values with those rates, which would either strengthen or correct the calibration claim.
Editorial extension: because the agent is a larger Llama 3 70B model, the better calibration may come partly from the agent's own reasoning ability rather than from the uncertainty-aware aggregation per se; a controlled comparison swapping the agent for a smaller or different model would separate those effects.
Editorial extension: the accuracy/F1 trade-off (Agent accuracy 0.8022 versus MLP 0.9242 after filtering) suggests the agent's uncertainty filtering is tuned for recall; in screening applications where missed findings are costly this is reasonable, but in settings where false positives trigger procedures, precision would matter more.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this paper reports a real, useful experiment—agent-based aggregation of six-prompt BayesPE ensembles for extracting structured findings from 9,683 Hebrew radiology reports, with 512 manually annotated reports as the test bed. That is a genuinely needed contribution: Hebrew is under-resourced in clinical NLP, and the dataset is multicenter and real. The direction—pooling several semantically equivalent prompts and having an LLM judge their agreement and confidence—is sensible and the paper describes it clearly enough to replicate. The abstract’s headline numbers (agent F1 0.3967, then 0.4787 after filtering) are plausible and the improvement over single-prompt baselines is credible. Credit where due: this is not a toy demo; the manual annotation, the label selection criterion (Kappa > 0.7 for the MLP training), and the explicit exclusion of a tuning set show care.

The soft spots are real and one is load-bearing. The central comparison in Tables 1–4 varies two factors at once: the extraction backbone is Llama-3-8B for every method, but the agent uses Llama-3-70B as the decision model, while the three entropy baselines have no LLM aggregator at all. So the agent’s higher F1, recall, and uncertainty separation could come from the 70B model’s extra capacity, not from the proposed “agent-based uncertainty awareness.” That is not a minor detail; it is the paper’s title claim. A matched 70B entropy baseline or an 8B agent would fix it, but neither is reported. The calibration claim also rests on hand-assigned uncertainty values (0/0.5/1 for the five confidence levels) and on median uncertainty separation only—no ECE, no reliability diagram, no significance tests. The post-filtering comparisons are computed on different retained subsets, so the F1 gains across methods are not directly comparable. And in Table 4, the Linear model’s F1 is listed as 0.212 while its precision is 0.3938 and recall 0.5079; that F1 is mathematically impossible, so the table has an internal inconsistency that needs correction. The paper’s own limitations section mentions domain transfer and class imbalance but misses the scale confound and the missing calibration metrics.

Who is this for? Researchers building retrospective datasets from non-English clinical text, and anyone working on uncertainty-aware LLM extraction. It deserves a serious referee, but the referee should insist on a scale-matched control, proper calibration diagnostics, and corrected reporting. The empirical foundation is solid enough that these are addressable in revision rather than fatal.

Referee Report

4 major / 6 minor

Summary. The paper proposes an agent-based uncertainty-aware method for structured extraction from Hebrew radiology reports. A Llama 3-8B model is queried with six semantically equivalent prompts; a Llama 3-70B agent then aggregates the prompt outputs into five confidence levels (Definitely/Likely Yes/No, Uncertain), which are mapped to uncertainty values 0, 0.5, and 1. This agent is compared against three entropy-based aggregators (uniform weights, linearly optimized weights, and an MLP weight predictor) on 462 manually annotated reports for 23 organ-finding labels. The paper reports that the agent achieves the best F1 and Kappa before filtering (F1=0.3967, Kappa=0.3006) and after removing high-uncertainty cases (F1=0.4787, Kappa=0.4258), and claims that the agent provides the best-calibrated uncertainty estimates.

Significance. If the claims were fully supported, the work would be a useful contribution to automated labeling of clinical text in a low-resource language, with a practical open-source pipeline and a genuine manual-annotation ground truth. The use of real radiology reports, manual labels by a radiologist, and a large multi-center corpus are strengths. However, the central comparison is currently confounded, and the calibration claim is not backed by the required evidence.

major comments (4)

[Methods: LLM utilization and Agent Decision Model; Tables 1-4] The main comparison conflates the aggregation mechanism with model scale. The entropy-based baselines (uniform, linear, MLP) operate directly on Llama 3-8B prompt outputs, whereas the agent uses Llama 3-70B as the decision model. Therefore the higher F1/Kappa and the uncertainty separation attributed to 'agent-based uncertainty awareness' could instead be explained by the greater capability of the 70B model. The paper must include matched conditions: an entropy-based aggregator applied to 70B outputs, or an agent implemented on the 8B backbone, or at minimum an explicit ablation quantifying the scale effect.
[Results: Uncertainty Histograms and Table 2] The claim that the agent is 'best-calibrated' is not supported by the evidence presented. Table 2 reports only the average median uncertainty for correct versus incorrect predictions; no calibration curve, expected calibration error, reliability diagram, or statistical test is provided. Moreover, the agent's uncertainty values are hand-assigned from five confidence categories (0, 0.5, 1) without validating the monotonic mapping between those categories and true error probability. The paper should report calibration metrics and, ideally, per-bin accuracy for the agent's discrete confidence categories.
[Results: Tables 3 and 4] The post-filtering comparisons are computed on different subsets for different methods: Table 3 excludes 43.31% (uniform), 20.97% (linear), 26.07% (MLP), and 33.16% (agent) of cases. Improvements after filtering are therefore not directly comparable, because coverage differs and each method is evaluated on the subset selected by its own uncertainty measure. The authors should report performance as a function of coverage or match exclusion rates across methods, and provide confidence intervals for the filtered metrics.
[Methods: Agent Decision Model] The procedure for resolving 'Uncertain' agent responses is underspecified. The text states that for the 'Uncertain' response, a final Yes/No decision is determined by 'aggregating the probabilities for each option from the previous step,' but the agent's inputs are defined as answers ('Yes' or 'No') and explanations, not probabilities. It is unclear where these probabilities come from (e.g., token logprobs, softmax over prompt votes, or something else) and how they are aggregated. Please clarify.

minor comments (6)

[Table 4] The Linear method's F1 of 0.212 in Table 4 is inconsistent with its precision (0.3938) and recall (0.5079), for which the harmonic mean is approximately 0.4436; this appears to be a typographical error and should be corrected.
[Methods: NLP Based Annotations] The manuscript states that retaining labels with Kappa above 0.7 'resulted in eight final labels,' but the evaluation section uses 23 organ-finding combinations; the relationship between the eight labels used for MLP training and the 23 labels used for evaluation should be stated explicitly.
[Abstract and Results] The abstract and conclusion state that the agent provides 'calibrated uncertainty predictions,' but the evaluation contains no calibration curve or ECE; the wording should be softened to 'better separation of median uncertainty' unless calibration metrics are added.
[Throughout] No confidence intervals or significance tests accompany any of the performance metrics in Tables 1-4; given the modest test set (462 reports) and 23 labels, the authors should report variability across labels or bootstrap intervals.
[Methods: Entropy-based Decision Models] The term 'Bayesian' in 'Bayesian Prompt Ensembles' is used loosely: the linear and MLP weighting schemes are deterministic optimizations, not Bayesian inference; the paper should either justify the terminology or use a more neutral term.
[Methods: Data Collection] The paper does not mention whether the code and model prompts are publicly available; for reproducibility of an 'open-source' pipeline, a code repository or data-sharing statement should be included.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central performance and calibration claims are evaluated against external manual annotations, and no fitted parameter is renamed as a prediction.

full rationale

The paper's main derivation chain is grounded in an external gold standard: 512 reports were manually annotated by a trained radiologist, and the reported F1, recall, and Kappa values are computed against those manual labels on a held-out test subset of 462 reports. The agent's uncertainty values (0, 0.5, 1) are hand-assigned to its confidence categories, but the claimed calibration is empirically checked by comparing median uncertainty for manually labeled correct versus incorrect predictions (Table 2), so correctness is not defined by the agent's own categories. The linear weights are tuned on 50 validation cases and the MLP is trained on automatically labeled data, but neither trained parameter is used as the final evaluation target; the held-out manual test set provides independent evidence. The self-citations to the in-house HSMP-BERT model ([17,18]) support the auto-labeling used for MLP training and label selection, but the agent method itself does not rely on those auto-labels for its predictions, and the agent comparison is evaluated on manual labels. The use of Llama 3-70B as the agent while entropy baselines use Llama 3-8B outputs is a potential experimental confound and a correctness risk, but it is not a circular reduction: the agent's superior numbers are not identical to its inputs by construction. No step in the paper equates a fitted parameter with a predicted quantity, and no load-bearing uniqueness or ansatz is imported solely through self-citation. Overall, the derivation is self-contained with respect to the manual ground truth, and no specific circular step can be quoted from the text.

Assumptions & free parameters 6 free parameters · 6 assumptions · 1 invented entities

The central comparison rests on several unvalidated domain assumptions: the reference standard is one radiologist, absent or non-visible organs are coded as negative, auto-labels train the MLP baseline, and the agent's discrete confidence mapping is treated as calibrated. The main fitted quantities are the baseline weight models and the filtering threshold; these are disclosed in outline but not fully specified.

free parameters (6)

Per-label linear prompt weights = not reported numerically
Fitted on 50 validation samples per organ-finding label by minimizing the paper's linear weight objective; used by the Linear baseline in all headline comparisons.
MLP prompt-weight network parameters = not released
Trained on 5,794 auto-labeled cases for the 8 Kappa>0.7 labels; used for the MLP baseline, but architecture, lambda, and exact training details are not given.
Uncertainty threshold for filtering = 0.5
Applied uniformly to all methods to define high-uncertainty exclusions; no threshold sweep or calibration-based selection is reported.
Entropy regularization coefficient lambda (MLP) = not reported
Appears in the MLP loss function; the value is chosen by the authors but not disclosed.
Agent confidence-to-uncertainty mapping = Definitely=0, Likely=0.5, Uncertain=1
Hand-assigned mapping; if incorrect, the reported calibration separation is unsupported.
Number of prompts = 6
Selected by authors based on marginally diminishing returns; a design hyperparameter that affects the ensemble and uncertainty estimates.

assumptions (6)

domain assumption Manual labels assigned by a single radiologist are treated as ground truth for all 23 organ-finding combinations.
Section 'Manual Data Annotation'; no inter-observer variability or second-reader adjudication is reported, so measurement error in the reference standard directly bounds all reported metrics.
domain assumption Labels 2 (organ absent due to surgery) and 9 (organ not visible on MRI) are collapsed to negative.
Methods: 'In the main experiment, we treated labels 2 and 9 as negative'; if absence or non-visibility is not equivalent to 'no finding', precision and recall estimates shift.
domain assumption HSMP-BERT auto-annotations with Kappa above 0.7 are reliable enough to train the MLP weight model.
Methods: 'we retained only labels with Cohen's Kappa score above 0.7', and the MLP is trained on auto-labeled cases; auto-label noise propagates into the MLP baseline.
ad hoc to paper The agent's five confidence categories map monotonically to true error probability with uncertainty values 0, 0.5, and 1.
Methods, Agent Decision Model; no calibration curve, ECE, or reliability diagram is provided, so this mapping is an unvalidated assumption underlying the 'calibrated' claim.
domain assumption Six ChatGPT-generated prompts are semantically equivalent and their diversity is sufficient for uncertainty estimation.
Methods and Appendix B; equivalence is asserted, not measured, and prompt selection itself is a manual design choice.
domain assumption A single random subset of 512 manually annotated reports, minus 50 for weight tuning, is representative enough to compare methods.
Methods, Data Collection and Evaluation Setup; no repeated sampling or cross-validation is reported.

invented entities (1)

Agent confidence levels (Definitely Yes, Likely Yes, Definitely No, Likely No, Uncertain)
purpose: Used to convert the aggregator's self-assessment into a discrete uncertainty score for filtering and calibration claims.
The five levels and their assigned uncertainties (0, 0.5, 1) are introduced by the authors; no external benchmark or calibration study independently validates that these categories correspond to the claimed error rates.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model." pith.science (2026). https://pith.science/paper/HI4BJEVC

@misc{pith2026250201691,
  author       = {Pith},
  title        = {Pith review of: Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HI4BJEVC}},
  note         = {Machine review of arXiv:2502.01691}
}

read the original abstract

Reliable extraction of structured data from radiology reports using Large Language Models (LLMs) remains challenging, especially for complex, non-English texts like Hebrew. This study introduces an agent-based uncertainty-aware approach to improve the trustworthiness of LLM predictions in medical applications. We analyzed 9,683 Hebrew radiology reports from Crohn's disease patients (from 2010 to 2023) across three medical centers. A subset of 512 reports was manually annotated for six gastrointestinal organs and 15 pathological findings, while the remaining reports were automatically annotated using HSMP-BERT. Structured data extraction was performed using Llama 3.1 (Llama 3-8b-instruct) with Bayesian Prompt Ensembles (BayesPE), which employed six semantically equivalent prompts to estimate uncertainty. An Agent-Based Decision Model integrated multiple prompt outputs into five confidence levels for calibrated uncertainty and was compared against three entropy-based models. Performance was evaluated using accuracy, F1 score, precision, recall, and Cohen's Kappa before and after filtering high-uncertainty cases. The agent-based model outperformed the baseline across all metrics, achieving an F1 score of 0.3967, recall of 0.6437, and Cohen's Kappa of 0.3006. After filtering high-uncertainty cases (greater than or equal to 0.5), the F1 score improved to 0.4787, and Kappa increased to 0.4258. Uncertainty histograms demonstrated clear separation between correct and incorrect predictions, with the agent-based model providing the most well-calibrated uncertainty estimates. By incorporating uncertainty-aware prompt ensembles and an agent-based decision model, this approach enhances the performance and reliability of LLMs in structured data extraction from radiology reports, offering a more interpretable and trustworthy solution for high-stakes medical applications.

Figures

Figures reproduced from arXiv: 2502.01691 by the authors.

**Figure 1.** (a) Distribution of organ-specific findings and (b) filtered organ-finding combinations with more than 15 positive cases [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

**Figure 2.** Illustration of the proposed Bayesian Prompt Ensemble pipeline for uncertainty-aware predictions. A radiology report and multiple semantically equivalent prompts are fed into an LLM, generating one prediction per prompt. These predictions are then aggregated to yield a final decision and uncertainty estimation. The aggregation function can be either an LLM agent or an entropy-based function that applies uniform weig… view at source ↗

**Figure 3.** Uncertainty Histograms for Ileum stenosis Computed by: (a) Uniform Weights, (b) Linear-Optimized Weights, (c) Learnable- MLP Weights, and (d) Agent-based decision [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Uncertainty Histograms for Ileum inflammation Computed by: (a) Uniform Weights, (b) Linear-Optimized Weights, (c) Learnable- MLP Weights, and (d) Agent-based decision [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

**Figure 1.** Figure 1: (a) Distribution of organ-specific findings and (b) filtered organ-finding combinations with more than 15 positive cases [PITH_FULL_IMAGE:figures/full_fig_p024_1.png]

**Figure 3.** Figure 3: Uncertainty Histograms for Ileum stenosis Computed by: (a) Uniform Weights, (b) LinearOptimized Weights, (c) Learnable- MLP Weights, and (d) Agent-based decision [PITH_FULL_IMAGE:figures/full_fig_p024_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 29 canonical work pages

[1]

Hadas Ben-Atya (MS), Faculty of Biomedical Engineering, Technion - Israel Institute of Technology, Haifa, Israel

work page
[2]

Naama Gavrielov, Faculty of Biomedical Engineering, Technion - Israel Institute of Technology, Haifa, Israel

work page
[3]

Zvi Badash, Faculty of Data and Decision Sciences, Technion - Israel Institute of Technology, Haifa, Israel

work page
[4]

Gili Focht (MSc), Juliet Keidan Institute of Pediatric Gastroenterology Hepatology and Nutrition, Shaare Zedek Medical Center, The Hebrew University School of Medicine, Jerusalem, Israel

work page
[5]

Ruth Cytter-Kuint (MD), Pediatric Radiology Unit, Radiology Department, The Eisenberg R&D Authority, Shaare Zedek Medical Center, The Hebrew University of Jerusalem, Jerusalem, Israel

work page
[6]

Talar Hagopian (MD), Pediatric Radiology Unit, Radiology Department, The Eisenberg R&D Authority, Shaare Zedek Medical Center, The Hebrew University of Jerusalem, Jerusalem, Israel

work page
[7]

Dan Turner (MD PhD), Juliet Keidan Institute of Pediatric Gastroenterology Hepatology and Nutrition, Shaare Zedek Medical Center, The Hebrew University School of Medicine, Jerusalem, Israel

work page
[8]

Moti Freiman (PhD), Faculty of Biomedical Engineering, Technion - Israel Institute of Technology, Haifa, Israel. Corresponding author contact information E-mail address: moti.freiman@technion.ac.il Full address: Technion, Faculty of Biomedical Engineering, Silver Building, 3200001 Funding information The study was sponsored by the Leona M. and Harry B. He...

work page 2010

Show all 36 references

[9]

Decision

Agent Decision Model We propose an Agent Decision Model to consolidate the outputs of multiple prompts, derive a final decision, and quantify its uncertainty. This model synthesizes responses and explanations from an ensemble of prompts, producing a unified decision with a rat...
[10]

No". Explanation:

Entropy-based Decision Models Alongside our agent-based decision model, we implemented three entropy-based models for comparison. These models determine the final prediction and its entropy by applying different weighting methods to the prompt outputs, as follows: a) Uniform w...
[11]

Structured reporting in radiology: a systematic review to explore its potential

Nobel JM, van Geel K, Robben SG. Structured reporting in radiology: a systematic review to explore its potential. European Radiology. 2022:1-18

2022
[12]

Information extraction from multi -institutional radiology reports

Hassanpour S, Langlotz CP. Information extraction from multi -institutional radiology reports. Artificial Intelligence in Medicine. 2016;66:29-39

2016
[13]

Classification, ontology, and precision medicine

Haendel MA, Chute CG, Robinson PN. Classification, ontology, and precision medicine. New England Journal of Medicine. 2018;379(15):1452-62

2018
[14]

A roadmap for foundational research on artificial intelligence in medical imaging: from the 2018 NIH/RSNA/ACR/The Academy Workshop

Langlotz CP, Allen B, Erickson BJ, Kalpathy-Cramer J, Bigelow K, Cook TS, et al. A roadmap for foundational research on artificial intelligence in medical imaging: from the 2018 NIH/RSNA/ACR/The Academy Workshop. Radiology. 2019;291(3):781-91

2018
[15]

Large language models for structured reporting in radiology: past, present, and future

Busch F, Hoffmann L, Dos Santos DP, Makowski MR, Saba L, Prucker P, et al. Large language models for structured reporting in radiology: past, present, and future. European Radiology. 2024:1-14

2024
[16]

Quantitative evaluation of large language models to streamline radiology report impressions: A multimodal retrospective analysis

Doshi R, Amin KS, Khosla P, Bajaj SS, Chheang S, Forman HP. Quantitative evaluation of large language models to streamline radiology report impressions: A multimodal retrospective analysis. Radiology. 2024;310(3):e231593

2024
[17]

A scoping review of large language model - based approaches for information extraction from radiology reports

Reichenpfader D, Müller H, Denecke K. A scoping review of large language model - based approaches for information extraction from radiology reports. NPJ Digital Medicine. 2024;7(1):222

2024
[18]

Privacy-ensuring open- weights large language models are competitive with closed -weights GPT -4o in extracting chest radiography findings from free -text reports

Nowak S, Wulff B, Layer YC, Theis M, Isaak A, Salam B, et al. Privacy-ensuring open- weights large language models are competitive with closed -weights GPT -4o in extracting chest radiography findings from free -text reports. Radiology. 2025;314(1):e240895

2025
[19]

Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilities

Shmidman S, Shmidman A, Cohen AD, Koppel M. Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilities. arXiv preprint arXiv:240707080. 2024

2024
[20]

Combining Confidence Elicitation and Sample-based Methods for Uncertainty Quantification in Misinformation Mitigation

Rivera M, Godbout JF, Rabbany R, Pelrine K. Combining Confidence Elicitation and Sample-based Methods for Uncertainty Quantification in Misinformation Mitigation. arXiv preprint arXiv:2401.08694. 2024

2024 arXiv
[21]

Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

Xiong M, Hu Z, Lu X, Li Y, Fu J, He J, et al. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. arXiv preprint arXiv:2306.13063. 2024

2024 arXiv
[22]

Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models

Zhou K, Jurafsky D, Hashimoto T. Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models. arXiv preprint arXiv:2302.13439. 2023

2023 arXiv
[23]

Bayesian prompt ensembles: Model uncertainty estimation for black-box large language models

Tonolini F, Massiah J, Aletras N, Kazai G. Bayesian prompt ensembles: Model uncertainty estimation for black-box large language models. Amazon Science. 2024

2024
[24]

The rise and potential of large language model-based agents: A survey

Xi Z, Chen W, Guo X, He W, Ding Y, Hong B, et al. The rise and potential of large language model-based agents: A survey. arXiv preprint arXiv:230907864. 2023

2023
[25]

Development and validation of novel algorithms to identify patients with inflammatory bowel diseases in Israel: an epi-IIRN group study

Friedman MY, Leventer-Roberts M, Rosenblum J, Zigman N, Goren I, Mourad V, et al. Development and validation of novel algorithms to identify patients with inflammatory bowel diseases in Israel: an epi-IIRN group study. Clinical Epidemiology. 2018:671-81

2018
[26]

Consensus recommendations for evaluation, interpretation, and utilization of computed tomography and magnetic resonance enterography in patients with small bowel Crohn’s disease

Bruining DH, Zimmermann EM, Loftus Jr EV, Sandborn WJ, Sauer CG, Strong SA, et al. Consensus recommendations for evaluation, interpretation, and utilization of computed tomography and magnetic resonance enterography in patients with small bowel Crohn’s disease. Radiology. 2018...

2018
[27]

P269 Harnessing Natural Language Processing for Structured Information Extraction from Radiology Reports in Crohn’s Disease: A Nationwide Study From the epi -IIRN

Hazan L, Focht G, Gavrielov N, Reichart R, Friss C, Cytter Kuint R, et al. P269 Harnessing Natural Language Processing for Structured Information Extraction from Radiology Reports in Crohn’s Disease: A Nationwide Study From the epi -IIRN. Journal of Crohn’s and Colitis. 2024;1...

2024
[28]

Leveraging Prompt-Learning for Structured Information Extraction from Crohn’s Disease Radiology Reports in a Low -Resource Language

Hazan L, Focht G, Gavrielov N, Reichart R, Hagopian T, Greer MLC, et al. Leveraging Prompt-Learning for Structured Information Extraction from Crohn’s Disease Radiology Reports in a Low -Resource Language. arXiv preprint arXiv:240501682. 2024

2024
[29]

Llama 3 Model Card; 2024

AI@Meta. Llama 3 Model Card; 2024. Available from: https://github.com/meta - llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md

2024
[30]

Attention Is All You Need

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention Is All You Need. arXiv preprint arXiv:1706.03762. 2017

2017 arXiv
[31]

Supervised Fine -Tuning as Inverse Reinforcement Learning

Sun H. Supervised Fine -Tuning as Inverse Reinforcement Learning. arXiv preprint arXiv:2403.12017. 2024

2024 arXiv
[32]

Deep Reinforcement Learning from Human Preferences

Christiano PF, Leike J, Brown T, Martic M, Legg S, Amodei D. Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems. 2017

2017
[33]

Learning to summarize from human feedback

Stiennon N, Ouyang L, Wu J, Ziegler DM, Lowe R, Voss C, et al. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325. 2022

2009 arXiv
[34]

Training language models to follow instructions with human feedback

Ouyang L, Wu J, Jiang X, Almeida D, Wainwright CL, Mishkin P, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. 2022

2022 arXiv
[35]

ChatGPT (GPT-4); 2024

OpenAI. ChatGPT (GPT-4); 2024. Available from: https://chat.openai.com

2024
[36]

Answer":

Zeng F, Lyu Z, Li Q, Li X. Enhancing LLMs for Impression Generation in Radiology Reports through a Multi-Agent System. arXiv preprint arXiv:241206828. 2024. Figure captions: Fig. 1: (a) Distribution of organ-specific findings and (b) filtered organ-finding combinations with mo...

2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Hadas Ben-Atya (MS), Faculty of Biomedical Engineering, Technion - Israel Institute of Technology, Haifa, Israel

work page

[2] [2]

Naama Gavrielov, Faculty of Biomedical Engineering, Technion - Israel Institute of Technology, Haifa, Israel

work page

[3] [3]

Zvi Badash, Faculty of Data and Decision Sciences, Technion - Israel Institute of Technology, Haifa, Israel

work page

[4] [4]

Gili Focht (MSc), Juliet Keidan Institute of Pediatric Gastroenterology Hepatology and Nutrition, Shaare Zedek Medical Center, The Hebrew University School of Medicine, Jerusalem, Israel

work page

[5] [5]

Ruth Cytter-Kuint (MD), Pediatric Radiology Unit, Radiology Department, The Eisenberg R&D Authority, Shaare Zedek Medical Center, The Hebrew University of Jerusalem, Jerusalem, Israel

work page

[6] [6]

Talar Hagopian (MD), Pediatric Radiology Unit, Radiology Department, The Eisenberg R&D Authority, Shaare Zedek Medical Center, The Hebrew University of Jerusalem, Jerusalem, Israel

work page

[7] [7]

Dan Turner (MD PhD), Juliet Keidan Institute of Pediatric Gastroenterology Hepatology and Nutrition, Shaare Zedek Medical Center, The Hebrew University School of Medicine, Jerusalem, Israel

work page

[8] [8]

Moti Freiman (PhD), Faculty of Biomedical Engineering, Technion - Israel Institute of Technology, Haifa, Israel. Corresponding author contact information E-mail address: moti.freiman@technion.ac.il Full address: Technion, Faculty of Biomedical Engineering, Silver Building, 3200001 Funding information The study was sponsored by the Leona M. and Harry B. He...

work page 2010