{"id":"ef329094-c4a8-4ecf-acdc-57151a308024","arxiv_id":"2411.18940","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Rewriting real electronic health records with small LLMs produces synthetic pretraining data that improves clinical language models at a small token budget.","lead":"The authors prompt four small AI language models to rewrite real hospital discharge notes, then train clinical AI models on these rewritten texts. Even a small amount of this synthetic text improves medical language modeling and beats an older clinical model, suggesting a cheaper path to building clinical AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Augmentation gains may be an artifact of doubled tokens: no control trains the baseline on 40M original tokens.","rationale":"The reader's weakest assumption was that LLM rephrasing preserves clinical meaning, which the paper itself acknowledges as unmeasured in Section 5. That is an important safety concern, but the central quantitative claim is the empirical superiority of rephrasing as a pretraining recipe. The most load-bearing flaw in the evidence as written is that the augmentation experiments compare a 20M-token original baseline against a 20M original + 20M synthetic setting, so any perplexity improvement could be pure data-quantity effect. This is directly checkable with an additional 20M original-token control, and it would settle whether 'augmenting original clinical notes with synthetic corpora improves performances even at a small token budget' is true. The reader noted a fixed token budget as a strength of the setup; for augmentation, that budget is not fixed. This does not change the overall conditional verdict, but it should be added explicitly as a required control. No claim about author intent is implied; the issue is the experimental design as reported.","tokens_in":10457,"tokens_out":7817,"duration_ms":75240,"concrete_test":"Train the same tiny Llama model for 5 epochs on 40M tokens of original MIMIC-III discharge notes (the original 20M sample plus a fresh 20M sample from the same source period), using identical hyperparameters and evaluation. If this model reaches the same perplexity as the 'Augmented with Llama-3.1' (or other) runs in Figure 2, the reported augmentation benefit is fully explained by doubled data. For the BERT downstream setup in Section 4, run the equivalent control: pretrain on 40M original tokens before fine-tuning on MedNLI and i2b2 2010/2012, and compare against the Real+Synthetic rows in Table 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The augmentation claim in Figures 2-3 does not hold the total number of pretraining tokens constant. Section 2.3 fixes each synthetic corpus at 20M tokens and each original chunk set at 20M tokens; Section 3.2 then reports pretraining on 'both real and synthetic clinical notes,' and Section 4.1 says 'we use only 20M sampled tokens for both real and synthetic text.' Under the natural reading, an augmented run sees 40M tokens, exactly double the 20M-token baseline that yields perplexity 10.76 in Figure 2. The large drops to roughly 8.8-9.5 could therefore be explained by data quantity alone, not by any property of rephrased synthetic text. The abstract's headline that augmenting with synthetic corpora 'improves performances even at a small token budget' thus lacks a like-for-like data-volume control. The paper's own Section 5 flags semantic preservation as unmeasured, but that is a separate safety concern; the quantitative confound here is more immediate because it directly undermines the main empirical claim. If the authors in fact mixed real and synthetic data to keep the total at 20M tokens, the paper should state this explicitly, and the numbers should be re-reported with that protocol.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes generating synthetic clinical pretraining corpora by prompting four small instruction-tuned LLMs (<10B) to rephrase chunks of MIMIC-III discharge summaries. The authors evaluate the approach in two settings: (i) causal language modeling with a 110M Llama model, measuring perplexity on MIMIC-IV discharge summaries and radiology reports under a temporal train/test split, and (ii) masked language modeling from BioBERT, fine-tuned on MedNLI, i2b2 2010, and i2b2 2012. The reported results indicate that rephrased corpora yield lower perplexity than the from-scratch Asclepius synthetic corpora, that augmenting real notes with synthetic corpora improves perplexity relative to real notes alone, and that synthetically augmented BERT models outperform ClinicalBERT on the downstream tasks.","tokens_in":10654,"tokens_out":6416,"duration_ms":56520,"significance":"If the main effects survive appropriate control conditions, this work would offer a practical, data-efficient recipe for institutional clinical language model pretraining, a timely contribution given the privacy constraints on sharing electronic health records. The study has notable methodological strengths: the synthetic-only comparison holds the token budget fixed at 20M across LLMs and prompts; the same source chunks are fed to all LLMs, controlling for input variation; and the test sets incorporate a temporal shift (MIMIC-IV notes from 2014 onward) and a different note type (radiology reports), making the perplexity evaluation more realistic than a random split. The rephrasing idea is well motivated by prior work on web-data rephrasing and is extended here with medical prompts, multiple LLMs, and both decoder- and encoder-based pretraining. However, the augmentation claim is currently confounded by doubled training tokens, and the absence of uncertainty quantification weakens the granular comparisons across prompts and LLMs.","major_comments":[{"comment":"Please see above.","section":"Section 3.2 / Figure 2; Section 4.1 / Table 1"},{"comment":"See above.","section":"Section 3 / Figures 1-3 / Table 1"},{"comment":"See above.","section":"Section 3.1 / Figure 1"},{"comment":"See above.","section":"Section 5"}],"minor_comments":[{"comment":"There are several typos: \"syntatically diverse\" (Section 2.3), \"clincal notes\" (Section 3.2), \"identifical\" (Section 4.1), and \"Ascplepius\" in the Figure 1 caption. These should be corrected.","section":"Section 2.3, 3.2, 4.1, Figure 1 caption"},{"comment":"The sentence \"we did not show [the original chunk] here due to data agreement by MIMIC-III\" is vague; it should be clarified whether the original chunk is omitted because of the MIMIC-III data use agreement and whether a redacted version could be included for comparison.","section":"Appendix A"},{"comment":"The statement that \"Llama seems more likely to use a boilerplate\" is informal and unquantified; if this is intended to explain Llama's underperformance, it should be supported by an analysis of repeated patterns across generated chunks.","section":"Appendix A / Section 3.2"},{"comment":"Fine-tuning hyperparameters for the BERT models are not reported beyond the pretraining configuration (learning rate, epochs, batch size). These details are needed for reproducibility of the MedNLI and i2b2 results.","section":"Section 4.1"},{"comment":"The choice to initialize the decoder from a TinyStories-pretrained 110M model is not justified; the manuscript should state whether this initialization could affect the relative comparisons across synthetic corpora, even if all conditions share the same base model.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and useful question, and the controlled synthetic-only comparison is a solid starting point. However, the augmentation claim is not yet supported because the augmented runs double the total pretraining tokens, and the lack of error bars weakens the prompt/LLM ranking conclusions. I do not see grounds for rejection: the issues are fixable within the manuscript's scope. The authors should be given the opportunity to add an iso-token control (e.g., 40M original tokens), run repeated trials, add a from-scratch baseline with the same small LLMs, and include a semantic-fidelity evaluation. If the augmentation effect disappears under the iso-token control, the paper's claims would need to be substantially toned down, but that is an empirical outcome, not a pre-existing fatal flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part of this paper is the fixed 20M-token comparison: rephrasing MIMIC-III discharge notes with small open LLMs gives lower perplexity on MIMIC-IV than Asclepius's from-scratch ChatGPT-3.5 synthetic corpus, across most prompts and models. That is a real, new application of the Maini et al. rephrasing recipe, and the temporal train/test split plus consistent input chunks are honest design choices. The prompt-sensitivity observations (e.g., Qwen-2 prefers the medically adapted prompt, Mistral prefers the generic one) are plausible and worth reporting.\n\nThe soft spot is the augmentation claim in Figures 2 and 3. As far as the text says, the baseline is 20M original tokens (PPL 10.76), and the augmented runs are 20M original plus 20M synthetic, i.e., 40M tokens total. If that is the protocol, the drop to roughly 8.8-9.5 could simply reflect twice as much data, not any special property of rephrased text. Section 4.1 carries the same ambiguity: \"we use only 20M sampled tokens for both real and synthetic text\" reads like each is 20M, so the augmented BERT models also see 40M tokens. The paper needs an explicit statement of the total token budget per run and, ideally, a 40M-token original-only baseline. Without that, the abstract's \"improves performances even at a small token budget\" is not supported as stated.\n\nOther issues are more ordinary: no variance or significance numbers, Phi-3-mini excluded post hoc (though the stated reason is reasonable), and the Asclepius baseline is a larger, closed model, so the fixed-generation-budget comparison is not like-for-like in model cost. The privacy premise is unvalidated: the paper itself flags in Section 5 that semantic preservation and hallucination are unmeasured, and there is no PHI-leakage evaluation, which matters for the claimed institutional deployment scenario.\n\nBottom line: the rephrasing-versus-scratch result survives scrutiny and is a reasonable contribution. The augmentation half needs a token-matched control before the headline claim is credible. This deserves a serious referee, but the referee should demand the 40M-token control and some PHI-leakage assessment. I would bring it to reading group as a useful case study in a common token-budget confound.","headline":"Worth reading for the fixed-budget rephrasing comparison, but the augmentation claim is likely confounded by a doubled token count.","tokens_in":11208,"tokens_out":2699,"would_cite":false,"duration_ms":22527,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rephrasing real clinical notes with small LLMs yields synthetic pretraining corpora that outperform from-scratch synthesis.","keywords":["clinical language models","synthetic pretraining data","rephrasing","electronic health records","privacy-preserving NLP","LLM paraphrase","MIMIC-III","clinical NLP"],"falsifier":"Extract clinical concepts, medications, doses, negations, and temporal relations from the original and rephrased versions of the same notes and compare their distributions; if they diverge systematically, or if a blinded clinician review finds fabricated facts in the rephrased text, then the method's benefit is distributional mimicry rather than content preservation. A direct test would corrupt a set of original notes with a wrong medication, rephrase them, and check whether the pretrained model learns the corrupted fact.","tokens_in":10222,"feed_emoji":"🩺","tokens_out":4806,"duration_ms":40833,"temperature":0.7,"pith_summary":"Clinical language models need large amounts of clinical text, but privacy rules make real electronic health record (EHR) notes hard to collect and share. This paper proposes generating synthetic pretraining corpora by asking small instruction-tuned LLMs (under 10 billion parameters) to paraphrase real discharge summaries, rather than inventing clinical notes from scratch. Across causal language modeling perplexity and downstream fine-tuning tasks, models pretrained on the rephrased corpora outperformed models trained on synthetic notes generated without referencing real text, and augmenting real notes with rephrased corpora improved performance even with only 20 million tokens. The result suggests that privacy-preserving pretraining can be built on paraphrase rather than generation, and that the method can support institution-level or larger-scale clinical language model development.","feed_headline":"Rephrasing real EHR notes beats AI-synthesized clinical text for pretraining","feed_subtitle":"Using 20M tokens of paraphrased MIMIC notes, models beat ClinicalBERT on MedNLI and i2b2 with far less data.","key_machinery":"The central machinery is the rephrasing pipeline: real clinical notes are split into chunks of about 300 tokens, each chunk is sent to a small instruction-tuned LLM under one of three prompts, and the outputs form a synthetic corpus used to pretrain or augment clinical language models. The role of the LLM is constrained to paraphrase, preserving the structure and content of real notes while changing surface form, which contrasts with from-scratch synthesis that relies on the LLM's internal knowledge to generate clinical details. Prompt choice matters: Prompt 3, which explicitly asks the LLM to explain medical terms, can hurt perplexity but helps on downstream natural language inference. Combining synthetic corpora from different LLMs or prompts with real notes provides stable gains.","core_discovery":"On its own terms, the paper establishes that rephrasing, not fresh synthesis, is the better route to synthetic clinical pretraining text. Using discharge summaries from MIMIC-III as source material, the authors prompt four instruction-tuned LLMs (Llama-3.1 8B, Mistral-0.3 7B, Qwen-2 7B, and Gemma-2 9B) to rewrite chunks of roughly 300 tokens under three prompts: a general Wikipedia-style paraphrase, a medically worded paraphrase, and a medical paraphrase that additionally asks for explanations of medical terms. A 110M-parameter Llama pretrained on each 20M-token synthetic corpus reaches lower perplexity on held-out MIMIC-IV discharge summaries and radiology reports than a model pretrained on Asclepius-style synthetic notes generated from scratch, in most cases by a large margin. Combining real notes with rephrased corpora lowers perplexity further, and BERT models initialized from BioBERT and pretrained on real-plus-rephrased text beat the ClinicalBERT baseline on MedNLI and i2b2 2010/2012 while using roughly 20M tokens versus the full MIMIC-III corpus.","pith_inferences":["If rephrasing preserves clinical meaning, the privacy argument is strengthened, but the paper does not measure re-identification risk; a testable extension is adversarial membership inference on the rephrased corpus to see whether it leaks information about the original notes.","The prompt-3 result hints at a tradeoff between content-preserving paraphrase and knowledge injection: adding explanations helps tasks like NLI that benefit from enriched context, while hurting language-model perplexity; measuring factual consistency across prompts would clarify this tradeoff.","The method may transfer to other note types such as radiology reports, progress notes, or discharge summaries in other languages, especially those with high boilerplate, though the paper only generates from MIMIC-III discharge summaries and evaluates on discharge summaries and radiology reports.","Because the paper does not include a qualitative clinical review of rephrased notes, a clinician-annotated comparison of original versus rephrased notes for factual errors would be a direct next step."],"forward_implications":["Institutions can pretrain or fine-tune clinical language models on rephrased versions of their own notes, potentially sharing the models without sharing raw patient text.","A small token budget of 20M rephrased tokens is enough to improve perplexity and downstream task performance, making the approach feasible for institution-level use and scalable to larger corpora.","Combining synthetic corpora from different LLMs or prompts yields additional gains, suggesting a recipe for scaling generation by diversity rather than by a single generator.","Masked language models pretrained with real-plus-rephrased text outperform the widely used ClinicalBERT on MedNLI and i2b2 benchmarks while using far fewer tokens.","The mixed results for medically adapted and explanation-seeking prompts indicate that prompt design is a controllable factor in synthetic clinical pretraining data quality."],"supporting_citations":[{"why":"Supplies the rephrasing recipe for web corpora that this paper adapts to clinical notes.","marker":"Maini et al., 2024"},{"why":"Provides the Asclepius synthetic-from-scratch baseline that the rephrasing method is compared against.","marker":"Kweon et al., 2024"},{"why":"Source of real clinical notes (MIMIC-III discharge summaries) used as the input to rephrasing.","marker":"Johnson et al., 2016"},{"why":"Held-out MIMIC-IV test sets for perplexity evaluation with a temporal shift.","marker":"Johnson et al., 2023"},{"why":"ClinicalBERT baseline for downstream fine-tuning comparison.","marker":"Alsentzer et al., 2019"},{"why":"Provides the tiny Llama architecture used for decoder pretraining experiments.","marker":"Touvron et al., 2023"},{"why":"MedNLI benchmark for downstream natural language inference evaluation.","marker":"Romanov and Shivade, 2018"},{"why":"i2b2 2010 NER benchmark for downstream named entity recognition evaluation.","marker":"Uzuner et al., 2011"},{"why":"i2b2 2012 benchmark for downstream clinical event and temporal relation extraction.","marker":"Sun et al., 2013"}],"fun_headline_variants":["Rephrased EHR notes outperform AI-synthesized clinical text","Paraphrasing real notes beats fresh AI generation for clinical models","20M tokens of rephrased notes beat synthetic clinical text","Better clinical pretraining from rephrased, not AI-generated, notes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that LLM rephrasing preserves the clinical content and meaning of the original notes closely enough that the synthetic text is both safe and useful for pretraining; the paper explicitly flags that it does not measure subtle meaning shifts or hallucinations.","fun_headline_variants_meta":{"raw":{"variants":["Rephrased EHR notes outperform AI-synthesized clinical text","Paraphrasing real notes beats fresh AI generation for clinical models","20M tokens of rephrased notes beat synthetic clinical text","Better clinical pretraining from rephrased, not AI-generated, notes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00039,"raw_usage":{"total_tokens":2062,"prompt_tokens":961,"completion_tokens":1101,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":1025}},"tokens_in":577,"tokens_out":1101,"duration_ms":9217,"temperature":1.0,"reasoning_tokens":1025,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:42:55.615418+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Extract clinical concepts, medications, doses, negations, and temporal relations from the original and rephrased versions of the same notes and compare their distributions; if they diverge systematically, or if a blinded clinician review finds fabricated facts in the rephrased text, then the method's benefit is distributional mimicry rather than content preservation. A direct test would corrupt a set of original notes with a wrong medication, rephrase them, and check whether the pretrained model learns the corrupted fact.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the rephrasing recipe for web corpora that this paper adapts to clinical notes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Asclepius synthetic-from-scratch baseline that the rephrasing method is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ClinicalBERT baseline for downstream fine-tuning comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MedNLI benchmark for downstream natural language inference evaluation."}],"review_version":1}