{"id":"a7616770-478c-4252-8b9e-eaf638bb04de","arxiv_id":"2505.17095","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"With temperature set to zero and ten repeated runs, all tested LLMs kept semantic consistency above 96% on clinical note generation, while Llama 70B and Mistral Small had the best combined consistency and correctness.","lead":"This study measured how consistently 12 large language models generated clinical notes from doctor-patient transcripts when asked the same prompt ten times, and how close the notes were to expert-written notes. It found nearly all models kept the same meaning across runs, with Meta's Llama 70B and Mistral's Small model ranking highest, suggesting smaller open-weight models may be suitable for privacy-preserving local deployment in clinical settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'most reliable' ranking of Llama 70B over GPT-4o-mini is asserted from an unweighted, untested comparison of SC and SS; with overlapping IQRs and inconsistent numbers between Table 2 and Figure 3, the central claim is not yet established.","rationale":"I read the paper as a careful empirical benchmark whose most valuable contribution is the explicit separation of string-level consistency, semantic consistency, and semantic similarity in CNG. The setup, the use of a public benchmark, and the candid limitations section are real strengths. However, the central advertised finding is the model ranking and the deployment recommendation that follows from it. That finding is not secured by the reported analysis: there is no defined aggregation of SC and SS, no significance testing over transcripts, a prompt-template confound for the Llama family, and apparent numerical discrepancies between Table 2 and Figure 3. The reader's weakest_assumption targets BERTScore's clinical sensitivity; I agree that is a genuine validity issue, but the more immediate problem is that even taking BERTScore at face value, the data as reported do not establish that Llama 70B or Mistral Small outperform OpenAI models. A revision that releases per-transcript scores and applies paired significance tests, fixes the numerical inconsistencies, and states the aggregation rule would largely address the concern; hence the conditional verdict remains appropriate rather than a rejection.","tokens_in":11716,"tokens_out":6098,"duration_ms":53313,"concrete_test":"Obtain the per-transcript SC and SS scores (or the raw notes) for all 12 models, then compute a pre-specified joint reliability score (e.g., standardized mean of SC and SS, or a Pareto-dominance rule) and run paired bootstrap or permutation tests between Llama-70B, Mistral Small, and the two GPT-4o variants. If Llama-70B no longer ranks first under reasonable weights, or if the difference vs GPT-4o-mini is not significant (e.g., 95% CI crosses zero), the paper should weaken the 'most reliable' conclusion to 'comparable within measurement uncertainty'. As a secondary check, re-run BERTScore with a fixed, named checkpoint and reconcile Table 2 with Figure 3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline ranking (Section 4.3, Conclusion) rests on the assertion that Llama 3.1-70B is best 'considering both semantic consistency and semantic similarity', but no aggregation function or significance test is given. Taking Table 2 at face value, Llama-70B has SC=100 and SS=85.90±1.29, while GPT-4o-mini has SC=97.40 and SS=87.26±1.24 (the highest SS in the table). Whether Llama-70B is 'more reliable' therefore depends on an implicit weighting of stability against correctness that the paper never specifies. The IQRs overlap, and no paired test over the 112 transcripts is reported, so the observed 1.1-1.4 point SS differences may be noise. The cross-family comparison is also confounded by the different prompt template used for Llama models (Appendix C). In addition, the quantitative results are internally inconsistent: Table 2 lists Llama-70B SSmedian as 85.90±1.29, but Figure 3 lists 87.56±1.4; similar discrepancies appear for Claude Haiku (85.61 vs 86.01) and Claude Sonnet (86.52 vs 86.78). These inconsistencies suggest the headline numbers are not stable enough to support a 'most reliable' claim, even before questioning whether BERTScore captures clinically meaningful correctness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of 12 large language models (Anthropic, Meta, Mistral, OpenAI) for clinical note generation (CNG) from the aci-bench dataset. Using 112 doctor-patient transcripts and 10 iterations of the same prompt per transcript, it measures three reliability-related quantities: consistency rate (exact string equality), semantic consistency (BERTScore among a model's repeated outputs), and semantic similarity (BERTScore against expert ground-truth notes). The main reported findings are that all models achieve median semantic consistency above 96%, that three models achieve perfect consistency rate, and that Meta's Llama 3.1 70B and Mistral Small are the most reliable models overall, leading the authors to recommend local deployment of relatively small open-weight models for CNG to address data-privacy concerns.","tokens_in":11956,"tokens_out":5050,"duration_ms":46437,"significance":"The paper addresses a genuine gap: empirical evidence on the intra-prompt reliability of open-weight LLMs for clinical note generation. Its methodological strengths include the use of an external public benchmark with expert ground-truth notes, 112 transcripts, 10 repeated runs per transcript, documented deterministic settings for the models, and clear reporting of medians and interquartile ranges for each metric. If the central findings held, they would be practically useful for privacy-conscious local deployment of open-weight models. However, the headline ranking and the 'correctness' interpretation currently rest on statistical comparisons and BERTScore calibration that are not yet established, and there are internal inconsistencies in the reported numbers. These issues need to be addressed before the practical recommendation can be accepted.","major_comments":[{"comment":"The claim that Llama 3.1-70B is 'most reliable considering both semantic consistency and semantic similarity' lacks a defined aggregation rule and statistical support. In Table 2, Llama-70B has SC=100 and SS=85.90±1.29, while GPT-4o-mini has SC=97.40 and SS=87.26±1.24 and ChatGPT-4o has SC=97.52 and SS=87.01±1.33. With no weighting specified and with overlapping IQRs, the asserted ranking could be reversed under an equally plausible weighting of stability versus correctness. I recommend adding an explicit composite reliability index or a paired statistical comparison over the 112 per-transcript values (for example, Wilcoxon signed-rank tests on SC and SS separately, or a paired bootstrap of the rank ordering) before drawing the 'most reliable' conclusion in Sections 4.3 and 5.","section":"Section 4.3 and Table 2"},{"comment":"The numeric results are internally inconsistent between Table 2 and Figure 3. For example, the median semantic similarity for Llama 3.1-70B is reported as 85.90±1.29 in Table 2 but 87.56±1.4 in Figure 3; Claude Haiku is 85.61±0.97 versus 86.01±0.94; Claude Sonnet is 86.52±1.21 versus 86.78±1.17; Llama 3.1-405B is 86.72±1.95 versus 87.44±1.29; and Mistral Small is 85.72±1.71 versus 87.24±1.75. These discrepancies are too large to be dismissed as typographical noise and they directly affect the ranking analysis and the conclusion. The authors must reconcile the values in the table and the figure, and should state which numbers are authoritative. Without this, the headline claim is not stable.","section":"Table 2 versus Figure 3"},{"comment":"Correctness is operationalized solely as BERTScore against the expert ground truth, with no human expert evaluation. Section 6 itself acknowledges the absence of human evaluation, and the prior validation of BERTScore for CNG cited in Section 2 (Giorgi et al., 2023) uses a different setup with in-context learning and temperature 0.2. Because BERTScore can be insensitive to clinically meaningful errors, such as a missing medication or an incorrect dosage, the statement that generated notes are 'correct' or 'close to expert notes' is not yet established. At minimum, the authors should specify the HuggingFace checkpoint used, and provide a calibration of BERTScore against clinician judgment or entity-level accuracy on a subset of the data. This is load-bearing because correctness is one of the two axes on which the 'most reliable' ranking is built.","section":"Sections 3.3.2, 4.2, and 6"}],"minor_comments":[{"comment":"The prompt template for Llama models is modified to include the special Llama chat-format tokens. Please clarify whether the change is purely the required chat template or whether any other wording was altered; otherwise the cross-family comparison is confounded and the apparent superiority of Llama 70B over proprietary models may be an artifact of the prompt format.","section":"Section 3.2.1 and Appendix C"},{"comment":"The claim that intra-prompt stability 'may be measured using semantic consistency alone' is stronger than the evidence supports. Consistency rate and semantic consistency capture different aspects of reliability, and the paper itself notes the value of string equivalence as a strict measure. I suggest softening this claim or providing a quantitative argument for why CR can be dropped.","section":"Section 4.1"},{"comment":"There are minor typographical issues in Table 1: 'ChatGPT-40-mini' should be 'ChatGPT-4o-mini', and the use of 'LLama' in Figure 3 should be made consistent with 'Llama' elsewhere.","section":"Table 1"},{"comment":"The statement that 'no study exploring the reliability of open-weight LLMs in CNG was found' is a strong claim; please provide a brief summary of the search strategy or soften the claim to avoid overstating the novelty.","section":"Section 1 and 6"},{"comment":"The manuscript does not include a data or code availability statement. Given the reproducibility value of the experimental protocol (112 transcripts, 10 iterations, explicit model configurations), adding a link to the generation and evaluation code would be useful.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical core and a worthwhile practical question, but the central 'most reliable' claim is not yet supported by the reported statistics, and the internal discrepancies between Table 2 and Figure 3 undercut confidence in the numbers. If the authors add pairing-aware statistical tests, define the aggregation rule, reconcile the reported values, and soften the correctness claim to what BERTScore can actually support, a revised version may be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical benchmark that fills a gap — nobody had evaluated eight open-weight models on intra-prompt stability for clinical note generation with repeated runs. The protocol is described in enough detail to reproduce (barring missing code/snapshots), and the main qualitative findings — all models are semantically stable above 96% median, and open-weight models match proprietary ones on both consistency and similarity to expert notes — look robust.\n\nWhat's genuinely new: the coverage. Twelve models, ten iterations per transcript, 112 aci-bench transcripts, with a clean separation between string-level consistency, semantic consistency, and correctness. That's a real contribution to the clinical NLP reliability literature, and the authors are appropriately cautious in the limitations about needing human expert validation and clinical utility studies.\n\nThe soft spots are mostly about the headline claim. The paper says Llama 3.1-70B is 'most reliable' based on considering both semantic consistency and semantic similarity, but there is no aggregation rule, no significance test, and the IQRs overlap. Worse, the numbers in Table 2 and Figure 3 don't match: Llama-70B's SSmedian is 85.90±1.29 in the table but 87.56±1.4 in the figure; Claude Haiku and Sonnet show similar discrepancies. That suggests the reported values aren't stable enough to support a ranking. The different prompt template for Llama models adds a confound, and the BERTScore implementation (which HuggingFace checkpoint?) is unspecified, which matters for a clinical correctness claim.\n\nNone of this sinks the paper. The stability conclusion doesn't depend on the exact ordering, and the recommendation to consider local deployment of open-weight models is reasonable given the data. But the 'most reliable' framing should be softened to 'comparable' or supported with paired tests and a defined metric.\n\nThe citation pattern looks fine; the related work is appropriate and the authors correctly position their contribution against Kernberg et al. and the aci-bench literature. No code or model snapshots, which is a limitation for reproducibility but not a fatal one.\n\nWho this is for: anyone working on clinical note generation or LLM reliability evaluation in healthcare. It deserves a serious referee and a revision that fixes the reporting inconsistencies, adds or explicitly declines significance testing, and specifies the BERTScore configuration. I'd bring it to reading group as a case study in how to (and not to) rank models from overlapping distributions.","headline":"Useful benchmark of 12 LLMs for clinical note generation, but the 'most reliable' ranking is not supported by the numbers as reported.","tokens_in":12539,"tokens_out":2152,"would_cite":true,"duration_ms":17800,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that all 12 tested LLMs are semantically stable in clinical note generation, that open-weight Llama 3.1 70B and Mistral Small are the most reliable models, and that these can be locally deployed while preserving patient…","keywords":["clinical note generation","large language model reliability","intra-prompt stability","semantic consistency","BERTScore","open-weight LLMs","aci-bench","healthcare data privacy"],"falsifier":"Re-score a random sample of generated notes on the aci-bench subset using a clinical-domain embedding model and have two or more clinicians independently mark critical errors; if the clinician rankings of model quality differ from the BERTScore ranking, or if notes with BERTScore above 80 contain critical omissions like a missed medication, then the claim that semantic consistency and semantic similarity certify reliability for clinical note generation would be refuted.","tokens_in":11519,"feed_emoji":"🩺","tokens_out":10848,"duration_ms":76299,"temperature":0.7,"pith_summary":"The paper asks whether large language models can be trusted to write clinical notes from doctor-patient conversation transcripts. Twelve models from Anthropic, Meta, Mistral, and OpenAI were each run ten times on the same 112-transcript prompt, and the repeated outputs were scored for string identity, semantic consistency, and similarity to expert-written notes. The headline finding is that all models are semantically stable, with median semantic consistency above 96% for every model, so the same clinical meaning recurs even when wording differs. Meta's Llama 3.1 70B is rated the most reliable overall, followed by Mistral's Small model, and both open-weight models outperform the proprietary ones. The paper concludes that these smaller open-weight models can be deployed locally for clinical note generation, which keeps protected health information out of third-party APIs.","feed_headline":"Open-weight Llama 70B most reliable for clinical notes","feed_subtitle":"Llama 3.1 70B and Mistral Small outrank GPT-4o and Claude, backing local deployment.","key_machinery":"The load-bearing mechanism is a three-metric reliability protocol using ten repeated generations from the same prompt. Consistency rate (CR) counts how often two generated notes are string-identical, semantic consistency (SC) measures the average cosine similarity, via BERTScore, between all pairs of generated notes, and semantic similarity (SS) measures BERTScore between a generated note and the expert ground-truth note. BERTScore, which prior clinical note work found suitable for this task, is the device that turns 'same meaning' and 'correctness' into numbers, and the deterministic model settings (temperature 0, top_p 0, top_k 1 where available) are what make the repeated runs comparable as a test of stability rather than sampling noise.","core_discovery":"On the paper's own terms, the discovery is that intra-prompt reliability for clinical note generation holds at the semantic level. Exact string output varied hugely between models, from 0% to 100% median consistency rate, yet every model's median semantic consistency exceeded 96%, and five models reached 100% semantic consistency. Correctness, measured as median BERTScore similarity between generated notes and expert ground-truth notes, ranged from about 80 to 88 across models, which the paper notes is higher than a previously reported best BERTScore of 60.8 on the same dataset. Meta's Llama 3.1 70B combines perfect semantic consistency with the best joint position on stability and correctness, Mistral's Small model is second, and both sit above all proprietary models. The paper presents this as evidence that reliable, privacy-preserving clinical note generation is achievable with locally deployed open-weight models.","pith_inferences":["If BERTScore is accepted as the right proxy, the paper's results imply that the remaining barrier to clinical use of LLM note generation is correctness rather than stability, pointing future work toward clinical validation, human expert review, and error analysis instead of consistency tuning.","Because the BERTScore checkpoint is unspecified, the model ranking reported here may be checkpoint-dependent; recomputing the twelve models with a clinical-domain embedding model would test whether Llama 70B's lead over GPT-4o persists or narrows.","The paper's local-deployment recommendation presumes that the privacy and cost benefits of open weights outweigh the operational burden of self-hosting; for low-resource settings, the edge-size Llama 1B and 3B models that achieved perfect semantic consistency but lower semantic similarity could offer a practical tradeoff that the paper does not quantify.","Generalizing beyond the aci-bench role-play transcripts, the near-universal high semantic consistency suggests that other long-form clinical summarization tasks, such as discharge summaries or consultation notes in other languages, would also show high stability, though correctness would need to be evaluated separately."],"forward_implications":["If the reliability ranking is correct, healthcare providers can run Llama 3.1 70B or Mistral Small locally for note generation and avoid sending protected health information to proprietary API endpoints.","Because all models stayed above 96% semantic consistency, string equivalence is not a necessary condition for reliable note generation; semantic consistency alone can serve as the monitoring metric for intra-prompt stability.","The BERTScore values in the low-to-mid 80s across all models, well above the 60.8 earlier baseline, indicate that current LLM-generated notes are meaningfully closer to expert notes than earlier in-context-learning results on the same dataset.","Within the Anthropic, Meta, and Mistral families, smaller models matched or beat their larger siblings on stability, while correctness favored larger models, implying that model choice for clinical note generation involves a separate scale-stability tradeoff for each family.","The recommendation to deploy open-weight models locally directly supports compliance with data privacy regulations in healthcare settings, since submitted prompts to proprietary platforms may be retained for training."],"supporting_citations":[{"why":"Supplies BERTScore, the metric used to define semantic consistency and semantic similarity.","marker":"Zhang et al., 2020"},{"why":"Provides the aci-bench dataset, the 112 transcripts and ground-truth expert notes used for generation and correctness comparison.","marker":"Yim et al., 2023"},{"why":"The prior intra-prompt stability study of one proprietary LLM in CNG that this paper extends to twelve models from four developers.","marker":"Kernberg et al., 2024"},{"why":"Establishes BERTScore as the appropriate metric for clinical note generation and supplies the BERTScore 60.8 baseline compared in the discussion.","marker":"Giorgi et al., 2023"},{"why":"Supports BERTScore's suitability for consultation note generation and the importance of semantic over string-equivalence evaluation given stylistic variability.","marker":"Moramarco et al., 2022"},{"why":"Supplies the consistency-rate operationalization (raw response agreement) used as the string-equivalence measure.","marker":"Atil et al., 2024"},{"why":"Supplies the semantic-consistency (consistency rate) measure used for intra-prompt stability and documents that no LLM in prior work achieved perfect semantic consistency.","marker":"Zhao et al., 2024"}],"fun_headline_variants":["Semantic reliability holds even when LLM strings vary","Llama 70B and Mistral Small top clinical note consistency","Open-weight models outrank GPT-4o for clinical notes","LLMs write clinically sound notes with stable meaning","Semantic stability in clinical notes across model families"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study's load-bearing premise is that BERTScore, computed with an unspecified HuggingFace checkpoint, correctly captures both the sameness of meaning across repeated notes and the clinical accuracy of a note relative to an expert note; if that embedding-based score misses clinically decisive errors such as a missing medication or a wrong dosage, the reported reliability ratings and the model ranking may not reflect clinical safety.","fun_headline_variants_meta":{"raw":{"variants":["Semantic reliability holds even when LLM strings vary","Llama 70B and Mistral Small top clinical note consistency","Open-weight models outrank GPT-4o for clinical notes","LLMs write clinically sound notes with stable meaning","Semantic stability in clinical notes across model families"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1230,"prompt_tokens":983,"completion_tokens":247,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":168}},"tokens_in":599,"tokens_out":247,"duration_ms":2674,"temperature":1.0,"reasoning_tokens":168,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:24:40.818374+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score a random sample of generated notes on the aci-bench subset using a clinical-domain embedding model and have two or more clinicians independently mark critical errors; if the clinician rankings of model quality differ from the BERTScore ranking, or if notes with BERTScore above 80 contain critical omissions like a missed medication, then the claim that semantic consistency and semantic similarity certify reliability for clinical note generation would be refuted.","supporting_citations":[{"cited_title":"Gold, and Vishnu Mohan","cited_arxiv_id":null,"evidence_quote":"The prior intra-prompt stability study of one proprietary LLM in CNG that this paper extends to twelve models from four developers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes BERTScore as the appropriate metric for clinical note generation and supplies the BERTScore 60.8 baseline compared in the discussion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the semantic-consistency (consistency rate) measure used for intra-prompt stability and documents that no LLM in prior work achieved perfect semantic consistency."}],"review_version":1}