{"id":"746dc7d5-78e3-4bfb-a3bd-6016167177b2","arxiv_id":"2502.01691","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An agent-based confidence filter raised F1 and Kappa for LLM extraction of findings from Hebrew radiology reports, though absolute performance stayed modest.","lead":"Researchers tested a method that uses multiple differently worded prompts and an AI 'agent' to label Hebrew radiology reports for signs of Crohn's disease, with each answer assigned a confidence level. They report that filtering out low-confidence answers improves accuracy and reliability, but the gains are modest and the confidence scale is not formally validated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Agent-vs-entropy comparison is confounded by model scale: the agent is Llama-3-70B while all entropy baselines run on 8B outputs, so the claimed improvement may come from capacity, not from the agent mechanism.","rationale":"The reader's weakest assumption (calibration monotonicity) is real and should be fixed with ECE/reliability diagrams, but it is secondary: if the model-size confound is not resolved, even a perfect calibration analysis would not establish the central claim. The paper has useful elements: a real clinical dataset, manual annotation, explicit prompt sets, and clear filtering comparisons. However, the central claim as stated is not yet supported because the agent method differs from the baselines in both aggregation algorithm and model capacity. I would keep the reader's CONDITIONAL verdict: the flaw is addressable with a matched-backbone control, not fatal to the underlying research direction.","tokens_in":11523,"tokens_out":4585,"duration_ms":48104,"concrete_test":"Re-run Tables 1 and 3 with a matched control: (a) use Llama-3-70B to provide the per-prompt probabilities for the uniform/linear/MLP aggregators (or to provide a direct forced-choice answer per prompt), and (b) run the agent with Llama-3-8B as the decision model. If the agent's F1/Kappa lead over the entropy baselines collapses in either condition, the improvement is due to model scale rather than the agent-based uncertainty mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Tables 1, 3, and 4 varies two factors at once. The extraction backbone is Llama-3-8B for every method (Methods: LLM utilization), but the agent uses Llama-3-70B as the decision model (Methods: Agent Decision Model, 'we utilized the Llama 3-70B model as the agent'). The uniform/linear/MLP entropy baselines contain no LLM aggregator at all. Hence the agent's higher F1/Kappa and its uncertainty separation (Table 2) can be explained by the 70B model's greater instruction-following and reading ability rather than by the proposed uncertainty-aware aggregation. This is not a minor implementation detail: the paper's title and abstract attribute the gain to 'agent-based uncertainty awareness.' A reader cannot tell how much of the 0.3967-to-0.4787 F1 improvement is due to the agent mechanism and how much to model scale. The same confound also undermines the calibration comparison: Table 2 compares the 70B agent's hand-mapped confidence labels (0/0.5/1) against entropy numbers computed from 8B prompt probabilities. No matched 70B entropy or 8B agent condition is reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an agent-based uncertainty-aware method for structured extraction from Hebrew radiology reports. A Llama 3-8B model is queried with six semantically equivalent prompts; a Llama 3-70B agent then aggregates the prompt outputs into five confidence levels (Definitely/Likely Yes/No, Uncertain), which are mapped to uncertainty values 0, 0.5, and 1. This agent is compared against three entropy-based aggregators (uniform weights, linearly optimized weights, and an MLP weight predictor) on 462 manually annotated reports for 23 organ-finding labels. The paper reports that the agent achieves the best F1 and Kappa before filtering (F1=0.3967, Kappa=0.3006) and after removing high-uncertainty cases (F1=0.4787, Kappa=0.4258), and claims that the agent provides the best-calibrated uncertainty estimates.","tokens_in":11788,"tokens_out":3097,"duration_ms":33384,"significance":"If the claims were fully supported, the work would be a useful contribution to automated labeling of clinical text in a low-resource language, with a practical open-source pipeline and a genuine manual-annotation ground truth. The use of real radiology reports, manual labels by a radiologist, and a large multi-center corpus are strengths. However, the central comparison is currently confounded, and the calibration claim is not backed by the required evidence.","major_comments":[{"comment":"The main comparison conflates the aggregation mechanism with model scale. The entropy-based baselines (uniform, linear, MLP) operate directly on Llama 3-8B prompt outputs, whereas the agent uses Llama 3-70B as the decision model. Therefore the higher F1/Kappa and the uncertainty separation attributed to 'agent-based uncertainty awareness' could instead be explained by the greater capability of the 70B model. The paper must include matched conditions: an entropy-based aggregator applied to 70B outputs, or an agent implemented on the 8B backbone, or at minimum an explicit ablation quantifying the scale effect.","section":"Methods: LLM utilization and Agent Decision Model; Tables 1-4"},{"comment":"The claim that the agent is 'best-calibrated' is not supported by the evidence presented. Table 2 reports only the average median uncertainty for correct versus incorrect predictions; no calibration curve, expected calibration error, reliability diagram, or statistical test is provided. Moreover, the agent's uncertainty values are hand-assigned from five confidence categories (0, 0.5, 1) without validating the monotonic mapping between those categories and true error probability. The paper should report calibration metrics and, ideally, per-bin accuracy for the agent's discrete confidence categories.","section":"Results: Uncertainty Histograms and Table 2"},{"comment":"The post-filtering comparisons are computed on different subsets for different methods: Table 3 excludes 43.31% (uniform), 20.97% (linear), 26.07% (MLP), and 33.16% (agent) of cases. Improvements after filtering are therefore not directly comparable, because coverage differs and each method is evaluated on the subset selected by its own uncertainty measure. The authors should report performance as a function of coverage or match exclusion rates across methods, and provide confidence intervals for the filtered metrics.","section":"Results: Tables 3 and 4"},{"comment":"The procedure for resolving 'Uncertain' agent responses is underspecified. The text states that for the 'Uncertain' response, a final Yes/No decision is determined by 'aggregating the probabilities for each option from the previous step,' but the agent's inputs are defined as answers ('Yes' or 'No') and explanations, not probabilities. It is unclear where these probabilities come from (e.g., token logprobs, softmax over prompt votes, or something else) and how they are aggregated. Please clarify.","section":"Methods: Agent Decision Model"}],"minor_comments":[{"comment":"The Linear method's F1 of 0.212 in Table 4 is inconsistent with its precision (0.3938) and recall (0.5079), for which the harmonic mean is approximately 0.4436; this appears to be a typographical error and should be corrected.","section":"Table 4"},{"comment":"The manuscript states that retaining labels with Kappa above 0.7 'resulted in eight final labels,' but the evaluation section uses 23 organ-finding combinations; the relationship between the eight labels used for MLP training and the 23 labels used for evaluation should be stated explicitly.","section":"Methods: NLP Based Annotations"},{"comment":"The abstract and conclusion state that the agent provides 'calibrated uncertainty predictions,' but the evaluation contains no calibration curve or ECE; the wording should be softened to 'better separation of median uncertainty' unless calibration metrics are added.","section":"Abstract and Results"},{"comment":"No confidence intervals or significance tests accompany any of the performance metrics in Tables 1-4; given the modest test set (462 reports) and 23 labels, the authors should report variability across labels or bootstrap intervals.","section":"Throughout"},{"comment":"The term 'Bayesian' in 'Bayesian Prompt Ensembles' is used loosely: the linear and MLP weighting schemes are deterministic optimizations, not Bayesian inference; the paper should either justify the terminology or use a more neutral term.","section":"Methods: Entropy-based Decision Models"},{"comment":"The paper does not mention whether the code and model prompts are publicly available; for reproducibility of an 'open-source' pipeline, a code repository or data-sharing statement should be included.","section":"Methods: Data Collection"}],"recommendation":"major_revision","confidential_remarks":"The core idea is potentially useful, and the manual annotation set is a real asset. However, the main evaluation confound (8B entropy baselines versus a 70B agent) is serious enough that the central claim as currently worded is not established. The calibration claim also needs proper metrics. These issues are fixable within the manuscript's scope by adding matched-model ablations, calibration analysis, and coverage-matched filtering comparisons, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper reports a real, useful experiment—agent-based aggregation of six-prompt BayesPE ensembles for extracting structured findings from 9,683 Hebrew radiology reports, with 512 manually annotated reports as the test bed. That is a genuinely needed contribution: Hebrew is under-resourced in clinical NLP, and the dataset is multicenter and real. The direction—pooling several semantically equivalent prompts and having an LLM judge their agreement and confidence—is sensible and the paper describes it clearly enough to replicate. The abstract’s headline numbers (agent F1 0.3967, then 0.4787 after filtering) are plausible and the improvement over single-prompt baselines is credible. Credit where due: this is not a toy demo; the manual annotation, the label selection criterion (Kappa > 0.7 for the MLP training), and the explicit exclusion of a tuning set show care.\n\nThe soft spots are real and one is load-bearing. The central comparison in Tables 1–4 varies two factors at once: the extraction backbone is Llama-3-8B for every method, but the agent uses Llama-3-70B as the decision model, while the three entropy baselines have no LLM aggregator at all. So the agent’s higher F1, recall, and uncertainty separation could come from the 70B model’s extra capacity, not from the proposed “agent-based uncertainty awareness.” That is not a minor detail; it is the paper’s title claim. A matched 70B entropy baseline or an 8B agent would fix it, but neither is reported. The calibration claim also rests on hand-assigned uncertainty values (0/0.5/1 for the five confidence levels) and on median uncertainty separation only—no ECE, no reliability diagram, no significance tests. The post-filtering comparisons are computed on different retained subsets, so the F1 gains across methods are not directly comparable. And in Table 4, the Linear model’s F1 is listed as 0.212 while its precision is 0.3938 and recall 0.5079; that F1 is mathematically impossible, so the table has an internal inconsistency that needs correction. The paper’s own limitations section mentions domain transfer and class imbalance but misses the scale confound and the missing calibration metrics.\n\nWho is this for? Researchers building retrospective datasets from non-English clinical text, and anyone working on uncertainty-aware LLM extraction. It deserves a serious referee, but the referee should insist on a scale-matched control, proper calibration diagnostics, and corrected reporting. The empirical foundation is solid enough that these are addressable in revision rather than fatal.","headline":"Useful empirical study in a low-resource language, but the headline agent-vs-entropy comparison is confounded by model scale (70B agent vs 8B baselines), and one reported F1 number is internally inconsistent.","tokens_in":12351,"tokens_out":1228,"would_cite":false,"duration_ms":14910,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An agent that pools six prompt answers and issues verdicts as Definitely, Likely, or Uncertain outperforms entropy-based uncertainty aggregation for automated Hebrew radiology report labeling.","keywords":["Deep Learning","LLM","Uncertainty Estimation","Radiology Report","agent-based decision model","Bayesian prompt ensembles","Hebrew radiology","structured data extraction"],"falsifier":"Compute the actual error rate among predictions the agent labels Definitely, Likely, and Uncertain on the 462-report test set. If 'Definitely' predictions are not substantially more accurate than 'Likely' ones, or if discarding all cases with uncertainty ≥0.5 does not remove a disproportionate share of errors, the claimed calibration advantage is falsified.","tokens_in":11292,"feed_emoji":"🩻","tokens_out":7623,"duration_ms":67330,"temperature":0.7,"pith_summary":"Automated extraction of structured findings from free-text radiology reports is unreliable, especially in low-resource clinical languages like Hebrew, where models can be confidently wrong. This paper tries to establish that an agent-based decision model—an LLM that reviews six semantically equivalent prompt answers and their explanations and issues a verdict in one of five confidence categories—yields both better labels and better-calibrated uncertainty than entropy-based pooling of the same prompts. On 462 manually annotated reports spanning 23 organ–finding combinations, the agent reaches F1 0.3967, recall 0.6437, and Cohen's Kappa 0.3006; discarding cases it flags with uncertainty ≥0.5 raises F1 to 0.4787 and Kappa to 0.4258. The broader contribution is a path to trustworthy open-source LLM use in high-stakes medical text processing, with uncertainty histograms that separate correct from incorrect predictions.","feed_headline":"Agent-based uncertainty pooling lifts Hebrew radiology F1 by 47%","feed_subtitle":"Six-prompt ensembles plus a confidence-labeling agent beat entropy baselines and flag cases for review.","key_machinery":"The central mechanism is the Agent Decision Model: an LLM (Llama 3 70B) that ingests the six answers and explanations from the BayesPE prompt ensemble and issues a structured JSON verdict in five confidence levels, with uncertainty assigned by hand as 0 for Definitely, 0.5 for Likely, and 1 for Uncertain. The agent is designed to weigh response consistency, explanation clarity, and signs of ambiguity. This categorical self-assessment, rather than a weighted entropy formula, is what the paper credits for the calibration separation and the post-filtering gains.","core_discovery":"The authors report that uncertainty for LLM-based structured data extraction is better modeled by an LLM agent's categorical self-assessment than by entropy computed over prompt ensembles. The agent receives the six yes/no answers and free-text explanations produced by Llama 3.1 8B under six semantically equivalent prompts (BayesPE), and using Llama 3 70B as the agent, it synthesizes them into one of five decisions: Definitely Yes, Likely Yes, Definitely No, Likely No, or Uncertain. These map to uncertainty values 0, 0.5, and 1. The agent's average median uncertainty is 0 for correct predictions and about 0.5 for incorrect predictions, a wider separation than uniform, linearly optimized, or MLP-learned weights. The resulting F1 of 0.3967 (recall 0.6437, Kappa 0.3006) exceeds all entropy baselines, and excluding high-uncertainty cases raises F1 to 0.4787 and Kappa to 0.4258.","pith_inferences":["Editorial extension: the paper fixes the agent's uncertainty values at 0, 0.5, and 1 by hand; a natural next test is to estimate the empirical error rate within each of the five confidence categories and replace the hand-set values with those rates, which would either strengthen or correct the calibration claim.","Editorial extension: because the agent is a larger Llama 3 70B model, the better calibration may come partly from the agent's own reasoning ability rather than from the uncertainty-aware aggregation per se; a controlled comparison swapping the agent for a smaller or different model would separate those effects.","Editorial extension: the accuracy/F1 trade-off (Agent accuracy 0.8022 versus MLP 0.9242 after filtering) suggests the agent's uncertainty filtering is tuned for recall; in screening applications where missed findings are costly this is reasonable, but in settings where false positives trigger procedures, precision would matter more."],"forward_implications":["Filtering out cases the agent labels Uncertain or Likely (uncertainty ≥ 0.5) improves F1 from 0.3967 to 0.4787 and Kappa from 0.3006 to 0.4258, so selectively referring high-uncertainty cases to human readers is a viable workflow.","Open-source Llama 3.1 can extract structured findings from Hebrew radiology reports more reliably when its six prompt answers are pooled by an agent than by entropy weighting, which matters for privacy-sensitive clinical data.","The agent's uncertainty histograms separate correct from incorrect predictions better than the entropy baselines, meaning the confidence labels carry practical information about when to trust the model.","The method's recall advantage (0.6437 versus 0.3977 for the MLP baseline) means it finds more true positives, at the cost of lower precision and accuracy, so task-specific thresholds may be needed.","Because the prompts are semantically equivalent and the agent output is structured JSON, the same pipeline can be adapted to other organs, findings, and languages without retraining the backbone LLM."],"supporting_citations":[{"why":"Supplies the BayesPE prompt-ensemble method and its six-prompt recipe for uncertainty estimation.","marker":"[13]"},{"why":"Defines the Llama 3.1 8B instruct backbone LLM used for structured extraction.","marker":"[19]"},{"why":"Provides the EPI-IIRN multicenter dataset of 9,683 Hebrew radiology reports.","marker":"[15]"},{"why":"Introduces the HSMP-BERT model used to auto-annotate the full dataset and select the eight labels with Kappa above 0.7.","marker":"[17,18]"},{"why":"Surveys LLM-based agents and motivates the adaptive decision-making architecture.","marker":"[14]"},{"why":"Prior work on multiple interacting LLM agents for radiology report generation, which this paper extends to uncertainty quantification.","marker":"[26]"},{"why":"Used to design the six semantically equivalent prompts that form the prompt ensemble.","marker":"[25]"}],"fun_headline_variants":["LLM agent's own uncertainty beats entropy in radiology labeling","Agent-calibrated confidence boosts Hebrew radiology F1","Hebrew radiology F1 rises with agent-based uncertainty filter","Uncertainty-aware agent improves LLM radiology extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's calibration benefit rests on the assumption that the agent's five confidence categories track true error probability monotonically, with uncertainty values 0, 0.5, and 1 hand-assigned; no calibration curve or per-category error rates are reported.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent's own uncertainty beats entropy in radiology labeling","Agent-calibrated confidence boosts Hebrew radiology F1","Hebrew radiology F1 rises with agent-based uncertainty filter","Uncertainty-aware agent improves LLM radiology extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":3146,"prompt_tokens":1109,"completion_tokens":2037,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":725,"completion_tokens_details":{"reasoning_tokens":1969}},"tokens_in":725,"tokens_out":2037,"duration_ms":13938,"temperature":1.0,"reasoning_tokens":1969,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:28:33.081866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the actual error rate among predictions the agent labels Definitely, Likely, and Uncertain on the 462-report test set. If 'Definitely' predictions are not substantially more accurate than 'Likely' ones, or if discarding all cases with uncertainty ≥0.5 does not remove a disproportionate share of errors, the claimed calibration advantage is falsified.","supporting_citations":[{"cited_title":"Classification, ontology, and precision medicine","cited_arxiv_id":null,"evidence_quote":"Supplies the BayesPE prompt-ensemble method and its six-prompt recipe for uncertainty estimation."},{"cited_title":"Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilities","cited_arxiv_id":null,"evidence_quote":"Defines the Llama 3.1 8B instruct backbone LLM used for structured extraction."},{"cited_title":"Large language models for structured reporting in radiology: past, present, and future","cited_arxiv_id":null,"evidence_quote":"Provides the EPI-IIRN multicenter dataset of 9,683 Hebrew radiology reports."},{"cited_title":"A roadmap for foundational research on artificial intelligence in medical imaging: from the 2018 NIH/RSNA/ACR/The Academy Workshop","cited_arxiv_id":null,"evidence_quote":"Surveys LLM-based agents and motivates the adaptive decision-making architecture."},{"cited_title":"Consensus recommendations for evaluation, interpretation, and utilization of computed tomography and magnetic resonance enterography in patients with small bowel Crohn’s disease","cited_arxiv_id":null,"evidence_quote":"Prior work on multiple interacting LLM agents for radiology report generation, which this paper extends to uncertainty quantification."},{"cited_title":"Development and validation of novel algorithms to identify patients with inflammatory bowel diseases in Israel: an epi-IIRN group study","cited_arxiv_id":null,"evidence_quote":"Used to design the six semantically equivalent prompts that form the prompt ensemble."}],"review_version":1}