Pith. sign in

REVIEW 4 major objections 5 minor 28 references

Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertise

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A process-supervised reward model built on an 8B open-source language model verifies clinical notes step-by-step, outperforming proprietary LLM judges on error detection and physician-preference selection.

desk verdict Useful, honest clinical NLP paper with a real new contribution—process supervision for note verification—but the headline 98.8% number is not yet trustworthy because the evaluation reuses the same synthetic error distribution used in training. read the letter →

arxiv 2412.12583 v3 pith:2UZTHDRV submitted 2024-12-17 cs.CL

Hanyin Wang , Chufan Gao , Qiping Xu , Bolun Liu , Guleid Hussein , Hariprasad Korsapati , Mohamad El Labban , Kingsley Iheasirim
show 4 more authors
Mohamed Hassan Gokhan Anil Brian Bartlett Jimeng Sun
This is my paper · ORCID
classification cs.CL
keywords process-supervisedrewardmodelclinicalnotegenerationbest-of-nselectionLLMverificationsyntheticerrorinjectionambientscribingassessmentandplanphysicianpreference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that process-supervised reward models can be built for clinical note generation, a domain without ground-truth answers, by defining clinically meaningful steps and using an LLM to inject realistic errors and paraphrases at scale. It reports that a PRM trained on LLaMA-3.1 8B with this synthetic process-supervision data distinguishes gold-standard from error-containing notes at 98.8% accuracy on an out-of-distribution task, and selects physician-preferred notes at 56.2%, beating proprietary models such as Gemini Pro 1.5, GPT-4o, o1, and o3-mini on the same tasks. A physician reader study indicates both evaluation metrics predict downstream Best-of-N quality, with preference selection more impactful. If correct, this provides a scalable automated alternative to costly clinician manual evaluation and a way to filter AI-scribe outputs before they reach a patient chart.

What carries the argument

The central object is the step-level score label: a special-token '+'/'−' prediction at each defined step of a clinical note, trained with cross-entropy loss on the note tokens (masking dialogue tokens), with the PRM score for a step being the softmax probability of the '+' token. The note-level score is the product (sum of logs) of all step-level scores, and the step hierarchy is defined by clinical documentation practice: each problem, each sentence in a problem, plus problem-level completeness, note-level completeness, and end-of-note steps. The synthetic data pipeline that injects physician-vetted error categories is what makes process supervision scalable without human step annotations.

What would settle it

Take a corpus of real clinical notes from a deployed ambient scribe, have clinicians mark the exact sentence or step containing an error, and measure whether the PRM's step scores identify those marked steps with comparable accuracy to the 98.8% reported on synthetic negatives; if real-data accuracy is substantially lower, the central claim that the PRM verifies clinical notes step-by-step is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that process supervision can be transferred from math and coding to open-ended clinical text by decomposing the Assessment and Plan section into a hierarchy of steps (each problem is a step, each sentence within a problem is a step, plus completeness steps at problem and note level and an end-of-note step), then training a reward model to predict a '+' or '-' label for each step. The training data is generated by prompting Gemini Pro 1.5 to produce errors in four physician-defined categories (factual inaccuracy, hallucination, unhelpfulness, incompleteness), swapping these into gold-reference notes, and adding paraphrases for diversity. The resulting PRM, built on LLaMA-3.1 8B, achieves state-of-the-art accuracy on the reported verification and preference-selection tasks, and the paper argues the data-generation model does not impose an upper bound because the PRM outperforms Gemini Pro 1.5 itself.

Load-bearing premise

The synthetic errors generated by Gemini Pro 1.5 under the four-category taxonomy are realistic, correctly labeled, and representative of the mistakes real LLM-written clinical notes actually contain, because all training negatives and evaluation negatives come from this same pipeline.

Editorial extensions

If this is right

  • A validated PRM can automate evaluation of AI-generated clinical notes, reducing reliance on costly and slow clinician reader studies, and can be reused when vendors update their scribe models.
  • PRM scores enable Best-of-N filtering at inference time, allowing an ambient scribe to present only the highest-quality note from many candidates.
  • Because errors are located at the step that caused the low score, the model offers explainable failure localization rather than a single pass/fail judgment.
  • The framework is model-agnostic for data generation and can be recalibrated for other medical specialties or other open-ended generative domains by redefining steps and error categories.
  • A-Prefer accuracy predicts physician-rated Best-of-N quality more strongly than A-Verify, so reward models targeting preference alignment are important for real-world deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the 98.8% verification number is measured against negatives created by the same synthetic error pipeline used for training, so real-world errors from deployed scribes may be more diverse or subtler and the headline accuracy likely overstates deployment performance until validated on human-flagged notes.
  • Beyond the paper: the 'each sentence is a step' heuristic is one of several reasonable granularities; a direct comparison of different step definitions would clarify whether step granularity is the actual driver of the PRM's advantage over outcome scoring.
  • Beyond the paper: the product-of-step-probabilities note-level score is an ad hoc aggregation, and the paper's own ablation table shows the choice matters; learned aggregation or thresholding could improve preference selection.
  • Beyond the paper: a natural extension is to use the step-level rewards as dense feedback for reinforcement learning on clinical note generators, which could push preference alignment beyond what Best-of-N filtering achieves.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes PRM-Clinic, a framework for training process-supervised reward models (PRMs) to verify LLM-generated clinical notes. The A&P section is decomposed into problem-level, sentence-level, completeness, and end-of-note steps; Gemini Pro 1.5 is used to inject errors from four categories (factual inaccuracy, hallucination, unhelpfulness, incompleteness) and to generate paraphrases, producing synthetic negative training samples. A LLaMA-3.1-8B model is trained with step-level score labels, then evaluated on verification tasks (A-Verify, Dialogue-G, A-Validate) and a physician-preference task (A-Prefer), compared against proprietary LLM judges and a vanilla ORM. The paper reports 98.8% accuracy on A-Verify and 56.2% on A-Prefer, along with ablation studies on loss functions and data selection and a physician reader study analyzing Best-of-N performance.

Significance. If the reported results hold, the paper would make a useful contribution: a compact open-source PRM that verifies clinical notes step-by-step could reduce reliance on expensive manual chart review for AI scribes, and the general framework of using LLM-generated process supervision guided by domain expertise would extend PRMs beyond math and coding. The release of code and dataset, the inclusion of a blinded physician reader study, and the systematic ablation of loss functions and data-selection strategies are concrete strengths. However, the central SOTA claim is currently supported only by evaluation tasks whose negative samples are generated by the same Gemini Pro 1.5 error-injection pipeline used to build the training set, so the 98.8% figure is evidence about recognition of the synthetic error distribution rather than about real clinical errors. The preference claim rests on 80 cases with no confidence intervals and involves checkpoint selection on the test set. With an independent real-error evaluation and more rigorous statistical reporting, the framework would be a valuable contribution to clinical NLP.

major comments (4)
  1. [§3.2, Table 2, Appendix Table 5] All three gold-reference verification tasks (A-Verify, Dialogue-G, and A-Validate) construct negative samples using the same four-category Gemini Pro 1.5 "Introduce Errors" prompt that was used to build the PRM-Clinic training set. Consequently, the reported 98.8% accuracy measures the model's ability to detect the synthetic error distribution produced by that specific prompt, not its ability to detect errors in naturally occurring or independently annotated clinical notes. The paper's own Sections 7 and 8 concede that the error scope is limited to four categories and that large-scale validation is needed, but no experiment in the manuscript quantifies transfer to real errors. I ask the authors to add a held-out evaluation with physician-identified or otherwise independently sourced errors, or to explicitly restrict the SOTA claim to synthetic error detection.
  2. [§4.1, Table 3] The A-Prefer result of 56.2% is based on 80 cases, which corresponds to 45 of 80 correct, and no confidence interval or significance test is provided against the 50.0% chance baseline or against o3-mini's 53.8%. Moreover, the model shown in Figure 3 is the checkpoint with the highest A-Prefer performance selected from the many Table 3 ablations on this same evaluation set, which is a form of selection on the test set. Please report confidence intervals and significance tests, and use a separate held-out set for model selection or otherwise correct for multiple comparisons.
  3. [§3.5, Table 2] A-Verify is described as out-of-distribution because the note generator changes from Gemini to LLaMA-Clinic, but the injected errors still come from the same Gemini Pro 1.5 pipeline used in training. This means the task does not test distribution shift in the error-generating process, which is the main generalization risk for a clinical verifier. The OOD claim should be justified with an independently generated error source, or the task should be renamed to reflect that only the note generator is out-of-distribution.
  4. [§3.4, Appendix C.2] The note-level PRM score is computed as the product (log-sum) of per-step softmax probabilities of the "+" token, and the paper asserts that this probability represents the chance that a step is correct. This calibration assumption is load-bearing for Best-of-N selection, but the paper does not validate calibration or examine whether the product scoring can be gamed by note length, step count, or stylistic artifacts. The physician reader study provides a partial check, but a calibration analysis (e.g., expected calibration error or reliability diagrams for step scores) and an ablation on normalized scoring would strengthen the claim that the step-level signals are meaningful.
minor comments (5)
  1. [§3.2, Table 1, §4.2] The dataset counts are inconsistent: Table 1 reports 9,680 total samples, while Section 4.2 states that the original dataset contains 10,952 samples. Please clarify whether Table 1 reports only negative samples or uses a different counting convention.
  2. [§4.1, Figure 3] Figure 3 shows accuracy values without error bars or confidence intervals, even for the small 80-case and 20-case evaluation sets; adding binomial confidence intervals would help readers assess the practical significance of the differences.
  3. [Appendix Table 9] The physician instructions allow ties in the A-Prefer task, but the paper does not explain how ties are handled when computing the 56.2% accuracy, which affects the chance level and the interpretation of the result.
  4. [Appendix D.2] The claim that other open-source models "could just as effectively" generate the synthetic errors is plausible but unsupported by experiments; if space permits, a small comparison would make the generality claim concrete.
  5. [References] Several references are incomplete or informal (e.g., "Beeching et al." has no year or title, and some arXiv entries lack version or venue information); please standardize the bibliography.

Circularity Check

2 steps flagged · score 4.0 of 10

Partially circular: the 56.2% A-Prefer headline is the test-set-selected maximum across checkpoints, and the 98.8% verification tasks reuse the same Gemini error-injection pipeline used to build the training data.

  1. fitted input called prediction [Table 3 note; Section 4.1; Abstract]
    "PRM and ORM results are reported from the checkpoint with the highest PRM A-Prefer performance. Numbers represent percentages of accuracy."

    The abstract and Section 4.1 present 56.2% A-Prefer accuracy as a key SOTA result. Table 3's note states that PRM/ORM numbers are taken from the checkpoint with the highest PRM A-Prefer performance, and A-Prefer is one of the two evaluation tasks (80 ACI-BENCH test2/test3 cases). Therefore the reported 56.2% is the value of the selection metric computed on the same test set used to choose the checkpoint; by construction it is the maximum of the evaluated checkpoints rather than an independent out-of-sample prediction. The headline 'prediction' is statistically forced by the reporting rule: optimize on the test set, then report the optimized value.

  2. other [Table 2; Section 3.2 'Introduce Errors']
    "Negative samples are introduced in the same manner as PRM-Clinic."

    PRM-Clinic training labels are created by prompting Gemini Pro 1.5 to generate four error categories and randomly swapping original steps with entries from the error pools. The verification tasks A-Verify, Dialogue-G, and A-Validate construct their negative samples with this exact same pipeline ('same manner as PRM-Clinic'), so the test negatives are draws from the same synthetic distribution the PRM was fit to. The 98.8% verification accuracy therefore measures detection of the signature of that specific injection procedure rather than verification of naturally occurring clinical errors.

full rationale

The core training loop is not an equation-level derivation: the PRM is trained with cross-entropy on step labels and evaluated on held-out cases, so there is no literal equality between the learned model and the test labels. However, the two headline claims are weakened by construction-related choices. First, the 56.2% A-Prefer figure is the maximum over checkpoints selected on that same test set, making it a selected statistic rather than an unbiased prediction. Second, the 98.8% verification numbers are measured on negatives produced by the same Gemini Pro 1.5 error-injection prompts and swap procedure used to create the PRM-Clinic training set (Section 3.2, Table 2); this demonstrates within-distribution detection of injected artifacts more than transfer to real clinical errors. The A-Prefer labels themselves come from a separate physician reader study in the authors' prior work (Wang et al., 2024a), which provides a genuinely external preference signal, and the test cases are not used for gradient updates, preventing full label leakage. The paper's own Section 7 concedes that the synthetic error scope should be expanded beyond the four categories, and Section 8 calls for larger validation. These issues are best characterized as benchmark-validity and selection-bias problems, but they make the paper's SOTA 'prediction' partially circular rather than fully independent.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on domain and data-generation assumptions: the Best Practice format and physician preferences come from prior work by the same group (Wang et al. 2024a); the step decomposition, error taxonomy, and label correctness are assumed on the basis of expert review; and Gemini Pro 1.5's outputs are treated as realistic proxies for real clinical errors. No new physical or theoretical entities are introduced; the new artifacts (PRM-Clinic dataset, trained checkpoint) are the results of the method rather than postulates. The free parameters are dataset-construction counts and a small set of training/scoring hyperparameters, none of which are the object of the paper's claims.

free parameters (5)
  • Errors injected per note = 10
    Prompt in Appendix Table 5 instructs Gemini Pro 1.5 to introduce 10 errors per note; this sets the training label distribution and the difficulty of the verification task.
  • Paraphrases generated per note = 20
    Prompt in Appendix Table 6 requests 20 paraphrases per note; controls semantic diversity of positive steps in training.
  • Learning rate = 3e-5
    Selected from {3e-4, 1.5e-4, 3e-5} in Appendix B.1; a hyperparameter choice that affects optimization.
  • Step-level score threshold for reader study selection = 0.5 (P('+') > P('-'))
    Appendix B.5: top-scoring candidates are chosen only if every step's '+' probability exceeds its '-' probability, mimicking Lightman et al. 2023; affects which notes physicians evaluate.
  • Sampling temperature grid for Best-of-2000 = 200 samples at each of 0.2-1.0, 100 at each of 1.1-2.0
    Appendix C.1: defines the candidate pool for the physician reader study; the distribution of top-10 samples concentrates at 0.4-1.0.
assumptions (5)
  • domain assumption The 'Best Practice' note format and the A&P-only focus, taken from Wang et al. 2024a, are the correct target for clinical note generation.
    Invoked in Section 3.1 and Appendix D.3; the prior work's physician panel defined the format, and the paper narrows to A&P because other sections are imported or less relevant.
  • domain assumption The four error types (factual inaccuracy, hallucination, unhelpfulness, incompleteness) cover the clinically relevant failure modes of LLM-generated notes.
    Section 3.2 'Introduce Errors'; identified through expert review of LLM notes, but completeness of the taxonomy is assumed, and Section 8 concedes other error types exist.
  • domain assumption Gemini Pro 1.5 generates errors that are realistic, correctly labeled, and usable as supervision.
    Section 3.2: error pools are generated via prompts and manually inspected by physician co-authors; the paper depends on the faithfulness of these labels.
  • domain assumption Physician preference labels from Wang et al. 2024a are a valid gold standard for note quality.
    Section 3.5 and Table 2: A-Prefer uses preference data from the first RLHF round of LLaMA-Clinic; the validity of those labels is assumed.
  • ad hoc to paper The softmax probability of the '+' token at each step is a calibrated correctness signal, and the log-product aggregation is a valid note-level score.
    Section 3.4 and Appendix C.2: the scoring choices are justified empirically (product wins on A-Prefer) rather than by an independent theory of reward calibration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertise." pith.science (2026). https://pith.science/paper/2UZTHDRV

@misc{pith2026241212583,
  author       = {Pith},
  title        = {Pith review of: Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertise},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2UZTHDRV}},
  note         = {Machine review of arXiv:2412.12583}
}
read the original abstract

Process-supervised reward models (PRMs) excel at providing step-by-step verification for large language model (LLM) outputs in domains like mathematics and coding. However, their application to fields lacking ground-truth answers, such as clinical note generation, poses significant challenges. We introduce a novel framework for training PRMs to deliver step-level reward signals for LLM-generated clinical notes. By precisely defining meaningful "steps," injecting realistic "errors" informed by domain expertise, and leveraging LLMs to generate process supervision data at scale, we overcome previous limitations. Our PRM, built on LLaMA-3.1 8B, consistently outperforms proprietary reasoning and non-reasoning models, achieving state-of-the-art performance on two key evaluations: (1) distinguishing gold-standard from error-containing samples with 98.8% accuracy, and (2) selecting physician-preferred clinical notes with 56.2% accuracy. We investigate critical components for effective PRM training, including optimal loss functions and data selection strategies, and present a comprehensive physician reader study identifying predictors of downstream Best-of-N performance. Our study sheds light on unlocking the potential of PRMs for diverse generative tasks across domains.

Figures

Figures reproduced from arXiv: 2412.12583 by the authors.

Figure 1
Figure 1. Overview of study design. We train a PRM to step-by-step verify clinical notes generated by LLMs from patient-doctor dialogues. Step 1: Clinical notes are transformed into a hierarchical structure of steps, designed based on domain expertise to capture the key considerations in clinical documentation. Step 2: Gemini Pro 1.5 is utilized to generate synthetic errors from predefined categories. These errors are systema… view at source ↗
Figure 2
Figure 2. Two notes to the same dialogue, graded by the PRM. In the top-scoring sample, the note is accurate and concise, with the PRM score (probability of a “+” score for a given step) exceeding 0.5 for each step. In the negative sample, various errors occur across steps, and PRM effectively assigns low scores to those erroneous steps. Items ending in _Score represent the PRM score assigned to each step. 3.2 Training Data C… view at source ↗
Figure 3
Figure 3. Main results. Our PRM outperforms all baseline models, including the vanilla ORM, Gemini Pro 1.5, and SOTA reasoning models, across all evaluated tasks. For A-Prefer, the task involves selecting the physician-preferred note from three candidate notes. For all other tasks, the objective is to identify the correct gold-reference sample from negative samples. For ID tasks, the notes are generated by Gemini Pro, while f… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Physician Reader Study Results. We se￾lected PRM checkpoints with varying A-Prefer and A￾Verify performances for group comparisons. When an￾alyzing the effect of a single metric, we identified two checkpoints with similar scores on the other metric. The Best-of-2000 no…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 20 canonical work pages

  1. [1]

    Assessment and Plan

    Consistent with prior research, the most frequent temperature range for high-scoring samples was found to be between 0.4 and 1.0. Appendix Figure 1: Distribution of top-10 scoring samples by PRM across temperature settings.This figure shows the counts of top-scoring samples per tem- perature setting for the physician reader study, with 2,000 samples gener...

    work page 2024

  2. [2]

    arXiv preprint arXiv:2412.16720

    Openai o1 system card. arXiv preprint arXiv:2412.16720. Jinhao Jiang, Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang, Yiru Tang, Haox- iang Sun, Jia Deng, Wayne Xin Zhao, et al

    arXiv

  3. [3]

    arXiv preprint arXiv:2411.11694

    Technical report: Enhancing llm reason- ing with reward-guided tree search. arXiv preprint arXiv:2411.11694. Qiao Jin, Fangyuan Chen, Yiliang Zhou, Ziyang Xu, Justin M Cheung, Robert Chen, Ronald M Summers, Justin F Rousseau, Peiyun Ni, Marc J Landsman, et al

    arXiv

  4. [4]

    ArXiv, pages arXiv–2401

    Hidden flaws behind expert-level accuracy of multimodal gpt-4 vision in medicine. ArXiv, pages arXiv–2401. Ron C Li, Trit Garg, Tony Cun, Lisa Shieh, Gomathi Krishnan, Daniel Fang, and Jonathan H Chen. 2018. Impact of problem-based charting on the utilization and accuracy of the electronic problem list. Journal of the American Medical Informatics Associat...

    arXiv 2018

  5. [5]

    Step_score

    Adding Ratings: - For each step, add a field called “Step_score” with the value “+”. - For each problem, add a field called “Problem_score” and “Problem_completeness_score” with the value “+”

    work page

  6. [6]

    Problem_no

    Add Sequential Numbering: - For each problem, add a field called “Problem_no”. Number them sequentially starting from “1”, and use strings (e.g., “"1"”, “"2"”). - For each step, add a field called “Step_no”. Number them sequentially starting from “1”, and use strings

    work page

  7. [7]

    Note_completeness_score

    Add Note Score: - Add a field called “Note_completeness_score” with the value “+” at the root level of the JSON

    work page

  8. [8]

    arXiv preprint arXiv:2403.05530

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530. Aaron A Tierney, Gregg Gayre, Brian Hoberman, Britt Mattern, Manuel Ballesca, Sarah B Wilson Hannay, Kate Castilla, Cindy S Lau, Patricia Kipnis, Vin- cent Liu, et al. 2025. Ambient artificial intelligence scribes: learnings after 1 year...

    arXiv 2025

Show all 28 references
  1. [11]

    Output Format: - The response should be in JSON format following the schema outlined below

  2. [12]

    Assessment and Plan

    Sections to Ignore: - Do not include any content labeled or titled “Assessment and Plan”

  3. [13]

    Follow-up instructions:

    Content Extraction: - Extract the remaining content of the clinical note and organize it into different problems. - Problems can be identified using a numbered problem list within the note. - When there’s a section titled “Follow-up instructions:”, treat this as a separate problem

  4. [14]

    Assessment:

    For Each Problem: - Extract each step as a separate sentence. - Ignore bullet point symbols, such as -, •, or other similar characters, when extracting steps. - If the words “Assessment:” or “Plan:” appear in the clinical note, include those words at the beginning of the first...

  5. [18]

    Problems

    JSON Schema: { “Problems”: [ { “Problem”: “Problem Description”, “Problem_no”: “1”, “Problem_score”: “+”, “Steps”: [ { “Step”: “First step of the problem.”, “Step_no”: “1”, “Step_score”: “+” }, { “Step”: “Second step of the problem.”, “Step_no”: “2”, “Step_score”: “+” } ], “Pr...

  6. [19]

    Assessment and Plan

    In each row, you will be given a synthetic outpatient patient-provider dialogue from Aci-bench, and two clinical notes based on the same dialogue. We will only evaluate the “Assessment and Plan” parts of a note

  7. [20]

    We have performed randomization of the notes and simple processing to unify the format of notes

  8. [21]

    Most conversations occurred in the outpatient setting

    The dialogues from Aci-bench include conversations with (a) calls to a virtual assistant, (b) unconstrained directions or discussions with a scribe, and (c) natural conversations between a doctor and patient. Most conversations occurred in the outpatient setting

  9. [22]

    Assessment and Plan,

    Since we are focusing solely on the “Assessment and Plan,” you may assume that all other pertinent information from the dialogue has been documented elsewhere in the note, which is not shown here. Please evaluate the “Assessment and Plan” as you would in a real note. For examp...

  10. [23]

    Assessment and Plan

    For each row, please start by reading the dialogue and then select your preferred notes. Make your selection based on the overall quality of the note. Essentially, choose the note you would prefer to use in a real patient encounter, imagining you are adopting AI-generated clin...

  11. [24]

    It’s perfectly acceptable to select a tie if you feel that two notes are equally good (or equally poor)

  12. [25]

    This is appreciated but not mandatory

    Please enter brief comments about each note to help us understand the rationale behind your selection. This is appreciated but not mandatory. Appendix Table 9: Instructions to physicians for note preference selection. Category Example Top-Scoring Sam- ple ASSESSMENT AND PLAN:

  13. [26]

    He experiences episodic shortness of breath, eye watering, and occasional diarrhea after heavy drinking

    Allergic Asthma Assessment: The patient has a history of allergic asthma, recently diagnosed by his primary care physician. He experiences episodic shortness of breath, eye watering, and occasional diarrhea after heavy drinking. Physical exam shows faint expiratory wheezing bi...

    2017

  14. [27]

    This suggests that she has chronic hepatitis C

    ASSESSMENT AND PLAN: Hepatitis C Assessment: The patient’s HCV Ab test was positive, and her liver panel showed an elevated AST (39), but her ALB, Tbili, and Total Bilirubin were all within normal limits. This suggests that she has chronic hepatitis C. Plan: - Confirm the diag...

  15. [28]

    She denies any other symptoms

    Fatigue Assessment: The patient reports feeling tired at the end of the day after working. She denies any other symptoms. Plan: - Recommend the patient to prioritize rest and relaxation. - Encourage the patient to maintain a healthy sleep schedule. - Consider referral to a sle...

  16. [2020]

    +” score label ex- ceeds that of its “ −

    and the AdamW optimizer (Loshchilov and Hutter, 2017). We adopted Huggingface’s Transformers library and utilized its Trainer mod- ule (Wolf et al., 2019). A limited search was con- ducted over learning rates [3e-4, 1.5e-4, 3e-5], with 3e-5 selected as the optimal value. The t...

    2017

  17. [2022]

    Advances in neural information processing systems, 35:27730–27744

    Training language models to follow instruc- tions with human feedback. Advances in neural information processing systems, 35:27730–27744. Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. 2024. O1 repli...

    2024 arXiv

  18. [2023]

    arXiv preprint arXiv:2305.20050

    Let’s verify step by step. arXiv preprint arXiv:2305.20050. Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou

    arXiv

  19. [2024]

    NEJM AI, 1(11):AIp2400392

    Preparing for the widespread adoption of clinic visit recording. NEJM AI, 1(11):AIp2400392. Emma Beavins. 2024. 2025 outlook: What’s next for ai scribes and virtual care. Fierce Healthcare, published December 18, 2024. Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling t...

    2024 arXiv

  20. [2025]

    arXiv preprint arXiv:2502.06703

    Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling. arXiv preprint arXiv:2502.06703. Tsai-Ling Liu, Timothy C Hetherington, Ajay Dharod, Tracey Carroll, Richa Bundy, Hieu Nguyen, Henry E Bundy, McKenzie Isreal, Andrew McWilliams, and Jeffrey A Cleveland....

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.