{"id":"d8d15458-651e-4563-8e0f-d32b50bffcee","arxiv_id":"2501.11199","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Embedding-driven diversity sampling of few-shot examples improves downstream classification with synthetic clinical text over random and zero-shot baselines on CheXpert radiology reports.","lead":"This paper tests whether selecting a diverse set of real clinical notes as few-shot examples improves the quality and utility of synthetic clinical text generated by a large language model. It reports that this diversity sampling outperforms random and zero-shot baselines on a radiology classification task, cutting the data needed to reach a performance target by roughly 40%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Diversity-vs-random comparison is confounded: the diversity condition's 50 baseline training notes are diversity-selected, while the random condition's baseline is random, so reported gains may reflect baseline composition rather than synthetic-data quality.","rationale":"The reader's weakest assumption concerned CLEAR label accuracy. That is a legitimate validity threat, but the experimental design already contains a more direct confound for the paper's central comparison. The diversity manipulation changes two things at once: (1) which 50 real notes are used as the initial training set, and (2) which notes are used as few-shot prompts for synthetic generation. Since Section 2.9 ties the baseline to the prompt set for the two few-shot conditions, the comparison between diversity sampling and random sampling cannot separate these effects. The '0.9 times as effective as real data' metric is particularly sensitive to the baseline because it measures how many augmentation points are needed to reach threshold; a better starting baseline reduces that count even if synthetic points are no better. A clean test is straightforward: hold the baseline fixed across conditions and vary only the augmentation source. I therefore keep the reader's CONDITIONAL verdict: the claim is plausible but not yet identified as due to diversity sampling per se, and the missing control should be required before acceptance. I disagree with the reader's choice of weakest assumption only in the sense that the baseline confound is more immediately decisive; the CLEAR-label concern remains secondary.","tokens_in":12227,"tokens_out":5930,"duration_ms":57001,"concrete_test":"Re-run the experiment with the 50 baseline training notes fixed to be identical across all augmentation conditions, e.g., use the same random 50-set as the baseline for diversity, random, zero-shot, and real-world augmentation, and separately repeat with the same diverse 50-set as the baseline for all conditions. Compare the incremental learning curves and 'additional data to reach 0.85' metric. If diversity-sampled synthetic augmentation still beats random-synthetic augmentation when both start from the same baseline, the central claim is supported; if the advantage disappears or reverses, the reported gains are attributable to baseline composition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.9 states that for the few-shot prompting experiments (diversity and random), the 50 baseline real notes used to fine-tune the classifier are the same 50 notes used for few-shot prompting, while for zero-shot and real-world sampling the baseline is randomly sampled. Consequently, the diversity condition begins from a k-means centroid-selected, intentionally diverse set of 50 real notes, whereas the random condition begins from a random 50-note set. Any advantage in learning efficiency (Table 3, Figure 2) or final AUROC/AUPRC (Table 4) could therefore be caused by the baseline being more informative, not by the synthetic notes generated from diverse prompts. The headline ratio 'one diversity sampled data point is approximately 0.9 times as effective as a real-world data point' (Table 5) compares augmentation points needed to reach 0.85 AUROC starting from different baselines (diverse vs random), so the ratio conflates baseline quality with synthetic-data quality. The full pipeline may still be useful, but the paper's causal attribution to diversity-driven synthetic data is not isolated.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an embedding-driven diversity sampling approach for few-shot synthetic clinical text generation. The method embeds real clinical notes with SFR-Embedding-Mistral, reduces dimensionality with UMAP, clusters with k-means, and selects 50 centroid-based diverse notes. These notes are used as few-shot prompts for LLaMA-3.1 8B to generate synthetic notes, which are then used to augment fine-tuning data for Bio+Clinical BERT on five CheXpert classification tasks. The authors compare against random few-shot sampling, zero-shot generation, and real-world sampling, reporting that diversity sampling improves synthetic data authenticity (cosine similarity and Turing test), reduces the number of augmentation steps needed to reach a 0.85 AUROC/AUPRC threshold, narrows the real-to-synthetic performance gap, and yields synthetic data that is approximately 0.9 times as effective as real data.","tokens_in":12475,"tokens_out":5601,"duration_ms":52348,"significance":"If the central claim is sound, the method offers a practical, low-compute way to bootstrap synthetic clinical text for fine-tuning when expert annotations are scarce. The downstream evaluation uses a manually annotated held-out test set, which is independent of the proposed method, and the comparison against multiple control conditions is a reasonable design. The Turing test with an expert clinician is a meaningful authenticity check. However, the paper's headline comparison between diversity and random sampling is confounded by the baseline selection procedure (Section 2.9), and the authenticity claim is not consistently supported by the reported numbers (Table 1). These issues are load-bearing for the paper's three stated contributions, so the present version does not yet establish the causal benefit of diversity sampling for synthetic data generation.","major_comments":[{"comment":"The diversity-vs-random comparison conflates baseline composition with synthetic-data quality. The text states that in the few-shot prompting experiments (both diversity and random), the 50 baseline real notes used for fine-tuning are the same notes used for prompting; for diversity these are k-means centroid-selected, while for random they are randomly sampled. Consequently, the diversity condition starts from a deliberately diverse and potentially more informative baseline, so the faster learning curves (Table 3, Figure 2), better final AUROC/AUPRC (Table 4), and the real-to-synthetic ratio (Table 5) may reflect baseline informativeness rather than the quality of the synthetic notes generated from diverse prompts. To isolate the effect of the synthetic data, the baseline should be held fixed across conditions (e.g., use the same 50 diverse notes as the fine-tuning baseline for both diversity and random prompting, or vary only the prompt-example selection).","section":"Section 2.9"},{"comment":"The claim that diversity sampling produces synthetic notes 'most similar to real-world data' is not supported by the reported numbers. Table 1 reports an average cosine similarity distance of 0.82 for both diversity sampling and zero-shot sampling, with random sampling at 0.73. The text says the diversity method was 'most similar' compared to zero-shot, but the table shows no difference on average. Please clarify the direction of the metric (whether lower or higher is better) and explain why zero-shot and diversity are equal; as written, the authenticity comparison does not differentiate diversity from zero-shot.","section":"Section 3.1 / Table 1"},{"comment":"The cosine-similarity authenticity evaluation uses the same embedding model (SFR-Embedding-Mistral) that is used to perform diversity sampling. Because the diversity selection explicitly optimizes for spread in this embedding space, measuring distance to real data in that same space risks favoring the diversity condition by construction. An independent embedding model (for example, Bio+Clinical BERT or a different sentence encoder) should be used for evaluation, or the authors should explain why this is not circular.","section":"Sections 2.4 and 2.8"},{"comment":"Label provenance is not specified consistently across conditions. The diverse baseline notes are labeled with the CLEAR pipeline (Section 2.5), and the real-world augmentation set is also labeled with CLEAR (Section 2.7), but the random sampling baseline and zero-shot baseline appear to use the original CheXpert agent labels from the working dataset without relabeling. If different label sources are used for different conditions, the comparison is confounded by label noise. Please specify the labeling procedure for every training subset and, ideally, report a validation of CLEAR's label accuracy on a sample of the working dataset.","section":"Sections 2.5, 2.7, and 2.9"}],"minor_comments":[{"comment":"The prompt-generation procedure for the random sampling control is not stated as explicitly as for diversity sampling; specify whether the same number of prompts (325 per class) and the same five-shot example sampling procedure were used.","section":"Section 2.5"},{"comment":"The term 'cosine similarity distance' is ambiguous. Define the formula (e.g., 1 - cosine similarity) and clarify whether larger or smaller values indicate greater similarity to real-world data.","section":"Table 1"},{"comment":"No information is given about the number of random seeds or repeated fine-tuning runs used to compute the reported means and 95% confidence intervals. Please state the number of runs and how the intervals were constructed.","section":"Figure 2 and Table 4"},{"comment":"The method for computing 'additional data to reach AUROC threshold of 0.85' is not described; clarify whether this is interpolation between the 25-point increments or from a fitted curve.","section":"Table 5"},{"comment":"The phrase 'one diversity sampled data point is approximately 0.9 times as effective as a real-world data point' conflates the number of augmentation points needed from different baselines with per-point effectiveness; please rephrase or provide a direct per-point marginal-effect analysis.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the downstream task evaluation is reasonably designed, but the reported comparison between diversity and random sampling is confounded by the baseline selection (Section 2.9). This is the main barrier to publication. The authenticity claim is also unsupported by Table 1 as currently presented. I would encourage the authors to re-run the experiments with a fixed baseline and to re-evaluate cosine similarity with an independent embedding model."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a practical paper with a real confound. The diversity-sampled baseline is not comparable to the random baseline, so the main claim about diversity-driven synthetic data isn't isolated.\n\nWhat's new: using k-means on UMAP-reduced embeddings to pick few-shot exemplars for generating synthetic clinical notes is a neat idea, and the downstream evaluation on CheXpert is reasonably thorough. The held-out test set is manually annotated, which avoids circularity for the utility evaluation. Final metrics have non-overlapping CIs. So there's a plausible method here.\n\nThe big problem is in Section 2.9: for diversity and random few-shot conditions, the 50 baseline notes used for fine-tuning are the same 50 notes used for prompting. For zero-shot and real-world, the baseline is simply random. The diversity baseline is deliberately selected from cluster centroids; the random baseline is random. So any gain in learning efficiency or final AUROC/AUPRC could come from the baseline being more informative, not from the synthetic notes. Table 5's '0.9x as effective' compares the number of augmentation points needed to hit 0.85 starting from different baselines, so the ratio conflates baseline quality with synthetic-data value. This is load-bearing because the paper's central claim is about diversity-driven synthetic data.\n\nAlso, the abstract says the approach produces notes that 'more closely align with real clinical text,' but Table 1 shows cosine similarity is the same for diversity and zero-shot (0.82). The Turing test does show diversity and random are hard to distinguish from real, while zero-shot is easy to spot, but it doesn't separate diversity from random. So the authenticity evidence is weaker than claimed.\n\nMinor: no error bars for the learning-rate step counts, no code/data release, single dataset, and the choices of k, UMAP hyperparameters, and 0.85 threshold look arbitrary.\n\nOverall: the paper is worth a serious referee, but it needs a redesign—either hold the baseline fixed across conditions or analyze the synthetic-data effect separately. As written, the headline conclusions outrun the evidence.","headline":"Useful idea, but the central diversity-vs-random comparison is confounded by different baseline sets, so the headline claims don't hold as stated.","tokens_in":12981,"tokens_out":3360,"would_cite":false,"duration_ms":31152,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diversity sampling of few-shot examples makes synthetic clinical notes almost as useful as real data for fine-tuning classifiers.","keywords":["diversity sampling","synthetic clinical text","few-shot prompting","CheXpert","large language models","data augmentation","embedding-driven selection","clinical NLP"],"falsifier":"Re-run the same pipeline with the 50 diverse examples labeled by manual clinician annotation instead of CLEAR's automatic labels; if the reported AUROC/AUPRC gains and the real-to-synthetic gap change materially, the results depend on CLEAR's label accuracy rather than on diversity sampling itself.","tokens_in":12052,"feed_emoji":"🩺","tokens_out":5352,"duration_ms":43700,"temperature":0.7,"pith_summary":"The paper claims that the way you pick the few real examples shown to a large language model matters as much as the generation itself. Selecting the 50 examples by embedding-driven diversity sampling—spreading them across the semantic space of clinical notes—produces synthetic chest radiology reports that are statistically indistinguishable from real ones and that train a classifier nearly as well as real data. The authors show that models fine-tuned with this synthetic data reach a 0.85 AUC with roughly 40% less data for AUROC and 30% less for AUPRC, and that augmenting with synthetic notes improves AUROC by 57% and AUPRC by 68% over random or zero-shot baselines. If correct, this gives a path to building clinical text classifiers when expert-annotated data is scarce.","feed_headline":"Diversity sampling makes synthetic notes 90% as useful as real data","feed_subtitle":"One diverse synthetic note is worth about 0.9 real ones, cutting training data to hit 0.85 AUC by up to 40%.","key_machinery":"The central object is the 'diverse data' set: 50 real clinical notes chosen as k-means centroids in a two-dimensional UMAP projection of SFR-Embedding-Mistral contextualized sentence embeddings. These notes serve as the five-shot examples in prompts to LLaMA-3.1 8B Instruct, steering the generator to reproduce the syntactic range of real radiology reports. The mechanism that carries the argument is the selection step: covering the embedding space with centroids rather than picking examples at random, so the generated notes span the same semantic variety as the real corpus.","core_discovery":"Diversity sampling produces synthetic clinical text that is closer to real notes in embedding space and in clinician judgment, and that delivers better downstream fine-tuning efficiency and final performance than random or zero-shot sampling. On CheXpert chest radiology reports, the approach generates notes that a clinical expert cannot reliably distinguish from real clinical text (52% correct synthetic identification, p=0.37), whereas zero-shot notes are easily spotted (96% correct, p<0.001). Fine-tuning Bio+Clinical BERT with these notes reaches the 0.85 AUROC threshold in 4.6 average steps versus 8.0 for random and 7.2 for zero-shot, and the final real-to-synthetic performance gap is 4.1% compared with 10.6% and 8.5% for the baselines. One diversity-sampled synthetic note is worth about 0.9 times a real note, a 60% improvement in value over random and zero-shot sampling.","pith_inferences":["The benefit likely comes from coverage of the embedding space rather than the specific embedding model, so the approach may transfer to other note types, languages, or clinical domains with any reasonable semantic encoder.","A practical workflow implied but not tested is to use automated labeling on the diverse 50 notes, generate synthetic notes for downstream tasks, and reserve expert annotation for a small one-time cost, reducing the annotation bottleneck.","The Turing test result suggests synthetic clinical text may be usable for privacy-restricted sharing or model development without exposing real patient notes, though the paper does not address de-identification risk.","The 0.9x effectiveness ratio is measured at a single AUC threshold; whether diversity sampling retains its advantage at higher performance ceilings or on multi-label tasks remains an open, testable question."],"forward_implications":["Reaching the 0.85 AUROC threshold requires 112 augmentation points with diversity sampling versus 177 for random and 176 for zero-shot, so less training data is needed.","The final performance gap to real-data training is 4.1% for diversity sampling versus 10.6% (AUROC) and 13.2% (AUPRC) for random and 8.5% and 12.9% for zero-shot, meaning synthetic data comes close to matching real data.","One diversity-sampled note is roughly 0.9 times as effective as a real note, versus about 0.56 for random and 0.57 for zero-shot, a large gain in data value.","A clinician cannot reliably tell diversity- or random-sampled synthetic notes from real clinical text, while zero-shot notes are clearly artificial, supporting the authenticity of the generated data.","Fine-tuning with diversity-sampled augmentation reaches the 0.85 AUROC cutoff in 4.6 average steps, close to the 4.0 steps of real-world data and well below the 8.0 and 7.2 steps of the baselines."],"supporting_citations":[{"why":"Supplies the CheXpert chest radiology reports and initial labels used for sampling, test sets, and the working dataset.","marker":"[30]"},{"why":"Provides the CLEAR automatic labeling pipeline that generates presence/absence labels for the diverse notes and real augmentation notes.","marker":"[32]"},{"why":"LLaMA-3.1 8B Instruct is the generator that produces synthetic clinical notes from the few-shot prompts.","marker":"[33]"},{"why":"Bio+Clinical BERT is the classifier fine-tuned on synthetic and real data to measure downstream performance.","marker":"[34]"},{"why":"UMAP performs the dimensionality reduction of embeddings used to cluster and select diverse examples.","marker":"[29]"},{"why":"MTEB supports the choice of SFR-Embedding-Mistral as the embedding model for diversity sampling.","marker":"[31]"},{"why":"Brown et al. few-shot learners underpins the few-shot prompting strategy that uses the diverse examples.","marker":"[28]"}],"fun_headline_variants":["Synthetic notes hit 90% of real data value via diversity","Diversity sampling: cut data 40%, keep AI accuracy","One diverse synthetic note equals 0.9 real ones","Embedding-driven sampling: 57% AUROC lift on synthetic data","Synthetic clinical text nearly matches real with diversity sampling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes the CLEAR automatic labeling pipeline produces accurate presence/absence labels for the 50 diverse notes and the 650 real-world augmentation notes; if those labels are systematically wrong, the synthetic notes and the augmented training data inherit the errors and the performance-gap measurements are biased.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic notes hit 90% of real data value via diversity","Diversity sampling: cut data 40%, keep AI accuracy","One diverse synthetic note equals 0.9 real ones","Embedding-driven sampling: 57% AUROC lift on synthetic data","Synthetic clinical text nearly matches real with diversity sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000924,"raw_usage":{"total_tokens":3961,"prompt_tokens":943,"completion_tokens":3018,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2932}},"tokens_in":559,"tokens_out":3018,"duration_ms":21966,"temperature":1.0,"reasoning_tokens":2932,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:31:48.295994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same pipeline with the 50 diverse examples labeled by manual clinician annotation instead of CLEAR's automatic labels; if the reported AUROC/AUPRC gains and the real-to-synthetic gap change materially, the results depend on CLEAR's label accuracy rather than on diversity sampling itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CheXpert chest radiology reports and initial labels used for sampling, test sets, and the working dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CLEAR automatic labeling pipeline that generates presence/absence labels for the diverse notes and real augmentation notes."},{"cited_title":"Meta AI https://ai.meta.com/blog/meta-llama-3/","cited_arxiv_id":null,"evidence_quote":"LLaMA-3.1 8B Instruct is the generator that produces synthetic clinical notes from the few-shot prompts."}],"review_version":1}