{"id":"af1af97e-6ce4-44e1-9a50-702477aaa380","arxiv_id":"2507.17754","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An EHR-integrated GPT-4o scribe with prompted Whisper transcription reached high adoption and self-reported burden reduction, but quality and burden claims rest on LLM judges and a non-randomized survey.","lead":"A telehealth company built an AI scribe that drafts SOAP notes from visit audio inside its electronic health record. Most surveyed clinicians said it lowered cognitive load and documentation burden, but the evidence is self-reported and lacks a control group.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's causal claim that the scribe reduces cognitive load and documentation burden rests entirely on a voluntary retrospective survey with no response rate, baseline, or control; the reported proportions describe respondents, not the 540-user population.","rationale":"The reader's weakest assumption is exactly the weak point I would stress. The central abstract claim is about causal reduction, so what must be true is that observed self-reports approximate true workload changes. This is least secure because the survey was voluntary, retrospective, uncontrolled, and single-item, and the workplace context creates demand characteristics. I agree with the conditional verdict rather than a harsher one: objective signals (70% VPC adoption, p50 14.4 s latency, and edit-rate/BERTScore comparisons at scale) give the system real plausibility, while the LLM-as-judge weakness is secondary because the burnout claim is the headline. The proposed check would settle whether the concern lands; if the response rate is high and balanced and EHR timestamps confirm time savings, the self-report evidence would be materially stronger. No internal inconsistency is present, just insufficient support for causal wording.","tokens_in":11298,"tokens_out":3227,"duration_ms":37094,"concrete_test":"Contact the corresponding author for the survey denominator and item-level response counts, then compare responders with non-responders on service line, scribe usage frequency, and note-completion times from EHR logs. If the response rate is low or skewed toward active users, or if there is no objective reduction in documentation time for a matched pre/post activation cohort, the causal phrasing should be replaced with a strictly self-report claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that self-selected retrospective survey responses measure actual workload reduction. Section 3.3 reports 94% (n=63) and 97% (n=66) agreement out of a population of over 540 clinicians, but no denominator, response rate, non-responder analysis, baseline, or control group is provided. The question wording asks whether the application 'reduced' burden, which requires comparison to pre-tool experience; in the absence of baseline measurement, the answers reflect subjective memory and are vulnerable to social desirability bias, particularly in a workplace where clinicians know the tool was built by colleagues. The two percentages also have different item-level denominators, suggesting non-response that is not discussed. Adoption data in Figure 4 show usage, not effect. The paper's own limitation section mentions only data privacy and employment, not this validity threat. This does not make the engineering story unsound, but it means the headline reduction claim is over-stated: the data support 'respondents reported reduced burden' and not 'the application reduces burden.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a production ambient scribe system deployed at Included Health. The system uses Whisper for transcription (with a TF-IDF-derived medical term prompt), a modular in-context learning pipeline with GPT-4o to generate SOAP sections and patient instructions, and a fine-tuned BART model to shorten the generated History of Present Illness text. The authors report that prompted Whisper reduces WER, that LLM judges prefer the ICL-generated SOAP notes over expert-written Primock57 notes, that over 540 clinicians adopted the system, and that surveyed clinicians self-report reduced cognitive load (94%, n=63) and reduced documentation burden (97%, n=66). They further report that BART post-processing reduces HPI length by 17% with only a 5% drop in BERTScore.","tokens_in":11492,"tokens_out":3819,"duration_ms":40013,"significance":"If the claims were fully supported, this would be a valuable real-world deployment study: it provides a concrete system architecture, adoption curves across service lines, latency data, and a practical post-processing approach to shorten AI-generated notes. The Primock57 WER comparison is a straightforward, testable result, and the production-scale adoption data are informative for the CS/HCI community. However, the central burden-reduction claim rests almost entirely on a voluntary, retrospective, uncontrolled survey with no response-rate reporting, and the note-quality claim relies on LLM-as-a-judge comparisons without human validation. These are load-bearing weaknesses because they support the title, abstract, and conclusion. The paper's engineering contribution is real but the evidence for its headline causal claims is currently insufficient.","major_comments":[{"comment":"The claim that the scribe 'reduces cognitive load and documentation burden' is supported only by a voluntary survey with no reported response rate, no baseline, no control group, and no non-responder analysis. The two survey items also have different denominators (63 and 66), which suggests item-level non-response that is not discussed. Since the items ask whether the application 'reduced' load or burden, the responses require retrospective comparison to pre-tool experience, and they are vulnerable to social desirability bias in a workplace where the tool was built by colleagues. The data support 'respondents reported reduced burden' but not 'the application reduces burden.' The abstract and conclusion should be re-scoped accordingly unless additional evidence (e.g., baseline workload measurements, response-rate information, or a comparison group) is provided.","section":"Section 3.3 and Abstract"},{"comment":"The LLM-as-a-judge evaluation reports win rates of 0.84 (GPT-4o), 0.97 (Claude 3.7-sonnet), and 1.00 (o1-preview) for ICL-generated notes over expert-written notes, but no human evaluation is provided. Because the generator is GPT-4o and the judges are also frontier LLMs with overlapping training data, the high win rates may partly reflect style self-preference or rubric-satisfying verbosity rather than clinical superiority. The claim that the notes 'exceed the quality of expert-written notes' would require either human expert validation or at least a demonstrated correlation between the LLM judges' preferences and human expert preferences.","section":"Section 3.2 and Table 2"},{"comment":"The BART post-processing evaluation does not state whether the 130,000-note fine-tuning dataset was split into training and test sets, nor whether the evaluation pairs in Table 3 overlap with training data. If the BART model was evaluated on notes it was trained to mimic, the reported 17% length reduction and 0.92 BERTScore are not trustworthy. Additionally, the authors acknowledge that post-processing eliminates some pertinent negatives, which conflicts with the 'little impact on semantics' claim when BERTScore is the only semantic metric; BERTScore with bert-base-uncased is a weak proxy for clinical adequacy.","section":"Section 2.4 and Section 3.4, Table 3"},{"comment":"The description of the Whisper prompt terms is ambiguous: the paper says the top-200 TF-IDF terms are extracted from an 'out-of-sample corpus of human-written Primock57 transcripts' but does not specify how this corpus is disjoint from the 32 test recordings. If the TF-IDF terms are estimated from transcripts of the same visits used for evaluation, the reported 19% WER improvement could be inflated by information leakage. The authors should clarify the exact split and how 'out-of-sample' is defined.","section":"Section 3.1"}],"minor_comments":[{"comment":"The row label 'GPT4o-transcribe-mini0.29 +/- 0.06' is missing a delimiter; the model name and WER value should be separated.","section":"Table 1"},{"comment":"The o1-preview row shows '1' without a standard deviation, unlike the other rows; the entry should be formatted consistently or an explanation should be given for the absence of a variance estimate.","section":"Table 2"},{"comment":"The rubric for Chronology contains a typographical error: 'the 0 0 - HPI needs improvement' should read 'the HPI needs improvement.'","section":"Appendix C"},{"comment":"The sentence 'Among respondents, 94% (n = 63) agreed that the application reduced their cognitive load during patient visits' would be clearer if the exact survey question wording and the full item text were included, because the interpretation of 'reduced' depends on the reference point.","section":"Section 3.3"},{"comment":"The Limitations section mentions data privacy and employment conflicts but omits the survey validity threats and the LLM-judge circularity discussed above; these should be acknowledged explicitly.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a genuine deployment case study, and the engineering details are useful. The main risk is that the title and abstract overstate the evidence: the survey is the only support for the burden-reduction claim, and it is not rigorous enough to carry that claim. I would advise the editor that a revision should either add meaningful evaluation (e.g., response-rate reporting, a baseline, or a minimal control comparison) or reframe the contribution as a system description with preliminary observational data. The LLM-judge result also needs either human validation or a carefully hedged interpretation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a solid production case study of an ambient scribe at Included Health, with real deployment data, but the abstract's causal claims about reducing cognitive load and documentation burden rest on a small voluntary survey, and the note-quality results come from LLM judges in the same model family as the generator. Treat it as engineering evidence, not as proof of effect.\n\nWhat is genuinely useful: over 540 clinicians used the tool; within three months it covered roughly 70% of virtual primary care visits and 40% of urgent care visits. The p50 latency of 14.4 seconds, adoption curves by service line, WER comparisons across Whisper and GPT-4o transcribe, and the 19% WER improvement from prompting Whisper with TF-IDF medical terms are concrete, reproducible-in-principle measurements. The BART post-processing result—17% length reduction with a 5% BERTScore drop—is a reasonable proof-of-concept, and the authors openly note that it removes some pertinent negatives. The system integration and modular prompting are not new methods, but they are described clearly enough for others to replicate the engineering.\n\nThe soft spots are real and concentrated in the headline claims. The survey in Section 3.3 has no reported response rate, no denominator, no baseline, and no control; the two percentages use different n's; and the questions ask respondents to compare retrospectively to their pre-tool experience. The data support \"survey respondents reported reduced burden,\" not \"the application reduces burden.\" The stress-test note is right about this. The LLM-as-a-judge win rates (0.84 to 1.0 against expert notes) are also weaker than they look: GPT-4o judges notes written by a GPT-4o pipeline, and there is no human validation or blinded clinician preference data. The TF-IDF prompt terms may come from the same Primock57 transcripts used for evaluation—the paper says \"out-of-sample\" but does not clarify the split—so that 19% WER gain should be treated with caution until the split is confirmed. The BART evaluation uses the clinician-edited notes as reference, which are also the training target; that inflates the apparent edit-learning success. The limitations section mentions data privacy and employment but not these validity threats. None of these flaws sink the deployment story, but they need to be fixed or the claims need to be softened before the abstract is credible. Citation pattern looks fine; relevant prior work is cited and competing interests are disclosed.\n\nThis paper is for health-system IT teams and researchers studying scribe deployment and evaluation methodology. It deserves a serious referee, but with a request for revision: survey response rate and non-responder analysis, human note-quality evaluation, a clear data split for the TF-IDF prompt, and an explicit discussion of the BART training-target issue. I would send it out.","headline":"A useful production case study with real adoption and latency data, but the abstract overstates the survey and LLM-judge evidence; send to peer review with demands for response-rate reporting, human evaluation, and data-split clarity.","tokens_in":12048,"tokens_out":2538,"would_cite":true,"duration_ms":26018,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The authors report that a custom-built AI scribe in a telehealth EHR reduced self-reported cognitive load and documentation burden for surveyed clinicians.","keywords":["ambient scribe","SOAP notes","telehealth","cognitive load","documentation burden","in-context learning","Whisper transcription","BART fine-tuning"],"falsifier":"An audit of the service line's EHR event logs could settle the burden claim: if clinicians who use the scribe spend no less time on documentation than their pre-rollout baseline, the 97% documentation-burden reduction is contradicted. For the quality claim, a blinded panel of practicing clinicians judging the same 32 Primock57 notes against the expert-written versions would provide a direct test of whether the LLM judges' preference for the generated notes reflects real clinical quality.","tokens_in":11076,"feed_emoji":"🩺","tokens_out":8086,"duration_ms":70702,"temperature":0.7,"pith_summary":"Inside a telehealth practice, the authors built an ambient scribe that records visits and automatically drafts SOAP notes and patient instructions, and they integrated it directly into the electronic health record clinicians already use. The paper's central claim is that the tool lightens clinicians' administrative load: 94% of surveyed clinicians said it reduced cognitive load during visits and 97% said it lowered documentation burden. Supporting evidence includes LLM-judge comparisons on mock visits where generated notes outperformed expert-written notes, production notes requiring only minor clinician edits, and a fine-tuned post-processor that shortens notes while mostly preserving meaning. If the claim holds, ambient scribes assembled from off-the-shelf speech recognition and large language models could ease a known driver of clinician burnout without forcing a workflow change.","feed_headline":"AI scribe cut charting burden for 97% of surveyed clinicians","feed_subtitle":"94% also reported lower cognitive load; generated notes beat expert notes in mock-visit judging.","key_machinery":"The load-bearing mechanism is a modular chain-of-thought in-context learning pipeline: rather than prompting one large language model call to write an entire SOAP note, the system sends three parallel prompts—Subjective/History of Present Illness, Past Medical Encounters and Vitals, and Assessment and Plan—plus a separate prompt for patient instructions, and then chains each generated subsection through a verification prompt. That totals six GPT-4o calls per visit. This modular design is what the paper credits for note quality, consistency, and the clinically useful features like documented pertinent negatives; the separate fine-tuned BART post-processor then compresses notes by learning clinician editing patterns. Whisper, prompted with domain terms, supplies the transcript input at a word error rate of 0.21 on mock medical audio.","core_discovery":"The authors report that a production ambient scribe—Whisper for transcription plus a modular in-context learning pipeline using GPT-4o for note generation—produces SOAP notes that, on 32 mock telehealth visits, were judged by multiple LLMs to be higher quality than the expert-written notes in the dataset (win rates 0.84 with GPT-4o judge, 0.97 with Claude 3.7-sonnet, and 1.0 with o1-preview). In live use, over 540 clinicians tried the tool within three months, and 94% (n=63) and 97% (n=66) of survey respondents respectively reported reduced cognitive load and documentation burden. Objectively, clinician-edited production notes were only about 4% shorter than the generated versions with a BERTScore F1 of 0.97, suggesting minor edits; fine-tuning BART on 130,000 pairs of generated and clinician-edited notes shrank the History of Present Illness section by 17% with a BERTScore drop to 0.92. These results are the paper's evidence that AI-generated documentation in telehealth can match or exceed human note quality while reducing the cognitive overhead of charting.","pith_inferences":["The paper's survey response rate and non-responder characteristics are not reported, so the 94% and 97% figures may overstate the effect; a randomized or time-motion comparison with objective documentation-time measurements would test whether perceived relief matches actual workload reduction.","The fine-tuned BART step's removal of pertinent negatives indicates a conciseness/completeness trade-off that could matter clinically; a clinician-preference study with the shorter notes would determine whether the compression is acceptable in practice.","The modular prompting design is not tied to SOAP in principle; the same parallel-chain-of-thought pattern could generate other structured documents, but the paper does not test that."],"forward_implications":["Clinicians using the scribe can produce SOAP notes that require only minor edits, since production notes differed from clinician-submitted versions by only 4% in length and 0.97 BERTScore F1.","Prompting Whisper with the top 200 TF-IDF medical terms from an out-of-sample corpus lowers word error rate on medical audio by 19%, which would likely transfer to other medical transcription deployments.","A fine-tuned BART post-processor can shrink AI-generated History of Present Illness sections by 17% while losing only 5% semantic similarity, suggesting a viable compression step before clinician review.","The modular parallel-subsection prompting strategy keeps note generation latency low enough for production use (p50 14.4 seconds), making real-time ambient scribing feasible in telehealth.","Widespread adoption within three months—nearly 70% of virtual primary care and 40% of urgent care visits—indicates clinicians will use such a tool when it is embedded in the existing EHR workflow."],"supporting_citations":[{"why":"Supplies Whisper, the transcription model whose output feeds the whole note-generation pipeline.","marker":"Radford et al., 2023"},{"why":"Supplies the Primock57 mock visits and expert-written notes used for WER and LLM-judge quality evaluation.","marker":"Korfiatis et al., 2022"},{"why":"Documents GPT-4o Transcribe, the comparison transcription model and the source of GPT-4o used for note generation.","marker":"OpenAI, 2025"},{"why":"Defines BART, the sequence-to-sequence model fine-tuned to compress generated notes.","marker":"Lewis et al., 2020"},{"why":"Provides BERTScore, the metric used to show semantic preservation after clinician edits and BART post-processing.","marker":"Zhang et al., 2020"},{"why":"Defines the SOAP note sections that the pipeline generates and that the evaluation rubrics score.","marker":"Podder et al., 2025"},{"why":"Quantifies time clinicians spend on EHR documentation, motivating the claim that reducing documentation burden matters.","marker":"Sinsky et al., 2016"},{"why":"Earlier ambient scribe deployment showing reduced documentation burden, the prior result this paper extends.","marker":"Tierney et al., 2024"},{"why":"Reports clinician experiences with ambient scribe technology, providing the comparative frame for the survey findings.","marker":"Duggan et al., 2025"}],"fun_headline_variants":["97% report less charting; AI notes beat experts","AI scribe: 97% less burden, notes beat experts","AI scribe cuts charting burden for 97% of clinicians","Telehealth AI scribe: 97% less charting, better notes","AI scribe outperforms experts, cuts burden 97%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes that the surveyed clinicians' retrospective self-reports—from a self-selected subset of over 540 users, with no baseline or control group—correspond to an actual reduction in cognitive load and documentation burden rather than to enthusiasm for a new tool.","fun_headline_variants_meta":{"raw":{"variants":["97% report less charting; AI notes beat experts","AI scribe: 97% less burden, notes beat experts","AI scribe cuts charting burden for 97% of clinicians","Telehealth AI scribe: 97% less charting, better notes","AI scribe outperforms experts, cuts burden 97%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00104,"raw_usage":{"total_tokens":4396,"prompt_tokens":988,"completion_tokens":3408,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":3318}},"tokens_in":604,"tokens_out":3408,"duration_ms":24421,"temperature":1.0,"reasoning_tokens":3318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:20:48.653632+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An audit of the service line's EHR event logs could settle the burden claim: if clinicians who use the scribe spend no less time on documentation than their pre-rollout baseline, the 97% documentation-burden reduction is contradicted. For the quality claim, a blinded panel of practicing clinicians judging the same 32 Primock57 notes against the expert-written versions would provide a direct test of whether the LLM judges' preference for the generated notes reflects real clinical quality.","supporting_citations":[],"review_version":1}