{"id":"879d8b19-960f-4968-b863-4ce91dc4c664","arxiv_id":"2509.01354","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A data-processing and privacy-preserving deployment framework claims state-of-the-art Chinese medical LLM accuracy and a 27% reduction in training-data leakage.","lead":"A Chinese medical LLM training pipeline that adds data cleaning, instruction generation, and preference filtering, plus a vector-database scheme that swaps model outputs for safe responses when a user query resembles training data. The authors report accuracy gains and a 27% reduction in a similarity-based privacy-leakage metric.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Continued pre-training corpus in Table 16 includes evaluation benchmarks (CMtMedQA, webMedQA, CMExam, CMB-Exam) with no stated test-split exclusion, so the SOTA/accuracy claims may reflect memorization rather than genuine improvement.","rationale":"The reader's weakest assumption is exactly the load-bearing one: the evaluation benchmarks appear in the continued pre-training corpus without any stated exclusion of test splits. I agree with this concern. The paper's internal ablations suggest the framework components have some effect, but all headline comparisons are on the same potentially contaminated benchmarks. The privacy reduction claim is also fragile—measured only on 1,812 pre-selected high-risk samples, reporting a drop in average similarity from ~0.85 to ~0.58 as a \"27% reduction\"—but the accuracy/SOTA claim is the primary advertised contribution and is the one most directly undermined by possible contamination. The proposed check is decisive: if overlap exists, the central claims are unsupported; if no overlap exists, the concern is resolved. Thus I keep the reader's REJECT verdict but emphasize this concrete verification step.","tokens_in":14798,"tokens_out":3637,"duration_ms":39633,"concrete_test":"Obtain the exact data splits used for continued pre-training (or hashes/IDs of every training sample) and the test sets for CMtMedQA, webMedQA, CMExam, CMB-Exam, PLE, C-Eval, and CMMLU. Compute exact overlap between training samples and evaluation items after normalization (for benchmarks, including question and answer options). If any overlap exists, retrain or at minimum recompute Tables 4 and 9 after removing all overlapping items from the training corpus, reporting accuracy with confidence intervals. If the DPF-CM vs. raw-data gaps (e.g., PLE 0.69 vs 0.62, CMB 0.77 vs 0.74, CMExam 0.73 vs 0.71) persist after decontamination, the SOTA claim survives; if not, it is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy/SOTA claim depends on the evaluation data being disjoint from the training data. Section 5.1 says the training mixture includes single-turn webMedQA, multi-turn CMtMedQA, and benchmarks CMB, CMExam, etc.; Table 16 lists CMtMedQA, webMedQA, CMExam, and CMB-Exam as continued pre-training corpora. D.1 says 'We use the test data to evaluate' only for webMedQA and says the CMtMedQA test set has 1000 items, but nowhere does the paper state that these exact test instances were excluded from the pre-training corpus. For CMExam and CMB-Exam, the paper does not even mention using test splits. If the pre-training corpus contained the same items or questions later scored in Tables 4, 9, and the ablations, then the model could answer from memorization, and the DPF-CM-specific improvements over the raw-trained baseline could shrink or vanish. This is a load-bearing correctness risk, not a stylistic issue: the abstract's claim of \"significantly improves model accuracy\" and \"state-of-the-art performance\" would be unsupported if contamination is present.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DPF-CM, a data processing framework for training and deploying Chinese medical LLMs, consisting of a training-data pipeline (continued pre-training, SFT, and preference-data denoising) and a deployment-time privacy-preserving vector database (PPVD) that identifies memorized training samples and replaces them with safe responses. The authors claim that DPF-CM significantly improves accuracy over an unprocessed baseline and achieves state-of-the-art results among open-source Chinese medical LLMs, while reducing training-data privacy leakage by 27%. The paper includes detailed data statistics, prompts, ablation studies, and comparisons with several baselines.","tokens_in":15098,"tokens_out":5011,"duration_ms":55688,"significance":"If validated, DPF-CM would be a useful holistic, data-centric contribution to Chinese medical LLM development, and its PPVD mechanism is a concrete engineering approach to mitigating training-data leakage at inference time. The paper is commendable for shipping a complete pipeline with reproducible prompts, data statistics, and extensive ablations. However, the central accuracy and SOTA claims rest on the assumption that evaluation data is disjoint from the training corpus, which is not demonstrated, and the privacy-leakage reduction is evaluated circularly. These are load-bearing validity concerns that must be resolved before the results can be accepted.","major_comments":[{"comment":"The continued pre-training corpus in Table 16 includes CMtMedQA, webMedQA, CMExam, and CMB-Exam, while Section 6 evaluates on CMtMedQA, webMedQA, CMExam, and CMB. The text only states for webMedQA that \"We use the test data to evaluate\"; for CMtMedQA it mentions a 1000-item test set but does not say these items were excluded from the 2.5M-token pre-training corpus, and for CMExam/CMB-Exam no test split is mentioned. If any evaluation questions appear in the training data, the accuracy gains in Tables 4 and 9 and the win rates in Tables 5-6 could reflect memorization rather than genuine improvement. This directly affects the abstract's \"significantly improves\" and \"state-of-the-art\" claims. The authors must demonstrate disjointness (e.g., exact and fuzzy deduplication of evaluation items against the training corpus) or re-evaluate on a truly held-out benchmark.","section":"§5.1 / Appendix D.1 / Table 16"},{"comment":"The claimed 27% privacy-leakage reduction is computed on the same 1,812 high-risk samples that were selected by the model-memory search and then protected. The protocol identifies samples with ROUGE-L > 0.85, applies the secure database, and re-measures average similarity on those same 1,812 samples. This is self-referential: it does not measure leakage on the other 98,181 samples or on non-flagged training samples, and it does not test adversarial prompts that may trigger memorization without being in the high-risk database. The privacy claim is therefore not established as a general property. Please evaluate on a held-out random sample of training data and on adversarial extraction or membership-inference prompts.","section":"§7.3"},{"comment":"No error bars, confidence intervals, or significance tests are reported. The paper repeatedly uses \"significantly\" (abstract and Section 6.1), but many benchmark differences are within one or two accuracy points (e.g., Table 4: CMMLU 0.79 vs 0.78; Table 9: PLE 0.69 for both Ours and GPT-4). Without variance or statistical testing, the significance claim is unsupported. Please report multiple seeds with standard deviations or bootstrap confidence intervals for the main comparisons.","section":"§6 / Tables 1-4, 9-12"}],"minor_comments":[{"comment":"The first sentence of Section 3 refers to \"PF-CMLT\" rather than \"DPF-CM\"; please correct.","section":"§3.1"},{"comment":"The CMtMedQA row appears twice with different numbers in the second and third columns; the table needs to be deduplicated and corrected.","section":"Table 5"},{"comment":"Only webMedQA is explicitly described as using the test split. Please state the exact evaluation split (train/test/exclusion) for every dataset used in Section 6.","section":"Appendix D.1"},{"comment":"The limitations section does not mention the potential overlap between training and evaluation data. Please add a discussion of how the benchmarks were kept disjoint from the training corpus.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The benchmark-contamination concern is the most serious issue; it is not an artifact of the review process and must be addressed before publication. The privacy evaluation is also circular. I recommend major revision rather than rejection because both issues are addressable with additional experiments or explicit analysis, but the paper is not acceptable in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a competent engineering write-up, but the headline numbers don't hold up as stated. The continued pre-training corpus in Table 16 includes CMtMedQA, webMedQA, CMExam, and CMB-Exam, and the evaluation in Sections 5.1 and 6 uses those same datasets. Nowhere does the paper say the test questions were held out. For webMedQA they mention using the test split, but that's not the same as excluding it from pre-training. For CMExam and CMB-Exam there's no mention of a test split at all. That is a load-bearing gap: without a clean split, the accuracy improvements and the SOTA comparison could just be memorization of answers. This is exactly the kind of contamination that invalidates an empirical claim, and the paper does nothing to rule it out.\n\nWhat does work: the chained-example instruction generation is a genuinely nice idea—linking examples in increasing quality order is a simple, reusable prompt-engineering trick. The ensemble reward-model denoising is sensible. And the PPVD deployment-time privacy scheme is thoughtful: splitting samples, identifying memorized content, and replacing with secure database responses is a reasonable blueprint. The ablation suite is thorough, even if some tables are messy.\n\nSofter spots: the privacy claim is narrower than advertised. The 27% reduction is measured only on the 1,812 high-risk samples that already exceeded the memorization threshold; the authors found those samples by the same ROUGE-L probe they then use to measure improvement. That's a circular evaluation on a self-selected subset. The attack model is also simple—a direct, known-prefix reconstruction—so the result doesn't generalize to adaptive adversaries. There are also no error bars or significance tests anywhere, which matters when the numbers are tight.\n\nWho should read it: people building Chinese medical LLMs and anyone interested in deployment-time privacy machinery. The core data-processing ideas are worth borrowing, but the empirical validation has to be redone. This paper deserves a peer review, not a desk reject, because the engineering is real and the contamination issue is fixable. The right verdict after review is major revision: either show explicitly that the evaluation sets were excluded from all training stages, or re-run the experiments with held-out data. If that can't be done, the SOTA claim should be withdrawn.","headline":"Likely train/eval overlap in the pre-training corpus undercuts the accuracy and SOTA claims; the engineering itself is plausible.","tokens_in":15548,"tokens_out":4833,"would_cite":false,"duration_ms":50080,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A data-processing pipeline lifts a 7B Chinese medical LLM to the top of open-source comparisons while cutting training-data leakage by 27%.","keywords":["Chinese medical LLM","data processing pipeline","chained example instruction generation","preference data denoising","reward model ensemble","privacy-preserving vector database","training data leakage","open-source medical benchmarks"],"falsifier":"Take the four datasets that appear in both the training corpus and the evaluation (CMtMedQA, webMedQA, CMExam, CMB-Exam), remove from the evaluation any question whose answer or phrasing appears in the training corpus (by exact match or high embedding similarity), and re-run the comparisons. If the accuracy margin against the open-source baselines largely vanishes after removing overlapped items, the SOTA claim is attributable to training-test leakage rather than to the data-processing pipeline.","tokens_in":14684,"feed_emoji":"🏥","tokens_out":4007,"duration_ms":46813,"temperature":0.7,"pith_summary":"This paper asks whether better processing of training data, rather than a new model architecture, can make a Chinese medical language model both more accurate and safer to deploy. It proposes DPF-CM, a two-module framework: one module cleans and augments data across continued pretraining, supervised fine-tuning, and preference-data selection; the other module, the Privacy Preserving Vector Database (PPVD), detects which training samples the model has memorized and replaces those responses during inference. The paper reports that a Qwen2.5-7B model trained with DPF-CM outperforms the open-source Chinese medical models it compares against, and that PPVD reduces the average similarity between model outputs and memorized training labels from about 0.85 to 0.58, a 27% reduction. If the evaluation results are taken at face value, the paper shows that data-centric and deployment-side interventions alone can produce large gains in specialized medical assistants.","feed_headline":"Data pipeline lifts Chinese medical LLM to SOTA and cuts leakage 27%","feed_subtitle":"Two cleansed data stages plus a vector database that swaps risky memorized replies for safe answers.","key_machinery":"The key mechanism on the training side is the chained-example instruction generation: seed question-instruction pairs are scored, sorted from low to high quality, and linked in a single prompt so the LLM learns a progressive refinement process and produces better instructions for new questions. The preference-data denoiser averages the 'preference distance' (reward-model score difference between chosen and rejected responses) across five independently trained reward models and drops samples whose distances are consistently too low or too high. On the deployment side, PPVD works by splitting training samples, using the first half as a prompt to elicit the model's memory, measuring output simi","core_discovery":"The paper's central claim is that a comprehensive data lifecycle—cleaning and generating continued-pretraining data, generating question-oriented instructions via a chained-example prompt, selecting and optimizing SFT data, and denoising preference data with an ensemble of reward models—yields a Chinese medical LLM that achieves state-of-the-art accuracy among same-size open-source Chinese medical models on medical dialogue, medical benchmarks, and medical terminology explanation tasks. A second claim is that the PPVD deployment module, built from a high-risk embedding database and a matched secure database, can intercept and replace memorized training responses during inference, cutting mea","pith_inferences":["If the evaluation benchmarks also appear in the continued-pretraining corpus, as Table 16 suggests for CMtMedQA, webMedQA, CMExam, and CMB-Exam, part of the reported SOTA margin might reflect memorization of test items rather than genuine processing gains; a clean held-out evaluation would settle this.","The 27% leakage reduction is measured only against a ROUGE-L similarity attack at a fixed threshold; other forms of attack such as paraphrase completion or membership inference could give different numbers.","PPVD's dependence on storing full embeddings for high-risk samples could be a scalability bottleneck for very large private corpora; weight-agnostic sketches or locality-sensitive hashes might achieve similar protection with less storage, as the paper itself flags as a limitation.","The chained-example instruction generation idea is a general data-augmentation technique that could be applied to other low-resource specialized domains beyond medicine, such as legal or financial question answering."],"forward_implications":["Data processing alone, without architectural changes, can move a 7B medical model to the top of open-source Chinese medical comparisons.","Chained-example instruction generation improves supervised fine-tuning more than generic few-shot prompting, especially for multi-turn medical dialogues.","Averaging multiple reward models and removing extreme preference distances improves downstream DPO model accuracy.","PPVD can reduce measured leakage of memorized training text from roughly 0.85 to 0.58 similarity while leaving ordinary response quality largely intact.","The framework transfers to a different base model (Ziya-LLaMA-13B-v1), suggesting the processing gains are not specific to the Qwen base."],"supporting_citations":[{"why":"Supplies the HuatuoGPT-II baseline and the huatuo26M single-turn dialogue dataset used in training and evaluation.","marker":"(Chen et al., 2024)"},{"why":"Supplies the Zhongjing baseline and the CMtMedQA multi-turn dialogue dataset used in training and evaluation.","marker":"(Yang et al., 2024b)"},{"why":"Supplies the webMedQA real-world Chinese medical QA dataset used for training and evaluation.","marker":"(He et al., 2019)"},{"why":"Supplies the CMB medical benchmark used to measure multiple-choice accuracy.","marker":"(Wang et al., 2024)"},{"why":"Supplies the CMExam medical exam benchmark used for evaluation and listed among continued-pretraining data.","marker":"(Liu et al., 2023)"},{"why":"Supplies the C-Eval benchmark, from which medical questions are used for evaluation.","marker":"(Huang et al., 2023)"},{"why":"Cited as the basis of the Minhash-LSH deduplication method used in SFT data processing.","marker":"(Bai et al., 2023)"},{"why":"Supplies GPT-4, used both as an AI judge for win/tie/loss evaluation and as a general LLM baseline.","marker":"(Achiam et al., 2023)"}],"fun_headline_variants":["New data pipeline boosts Chinese medical AI to SOTA, slashes leaks","Medical LLM hits SOTA with privacy-safe vector DB, cut leakage 27%","DPF-CM: Cleaner data, smarter Chinese medical LLM, 27% less leakage","How a dual-pronged framework made Chinese medical LLM SOTA and safer","Data framework lifts medical LLM to top ranks while cutting leaks 27%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The claimed accuracy gains assume the evaluation benchmarks were not also part of the training data; the paper lists CMtMedQA, webMedQA, CMExam, and CMB-Exam as continued pre-training data and does not state that test splits were excluded.","fun_headline_variants_meta":{"raw":{"variants":["New data pipeline boosts Chinese medical AI to SOTA, slashes leaks","Medical LLM hits SOTA with privacy-safe vector DB, cut leakage 27%","DPF-CM: Cleaner data, smarter Chinese medical LLM, 27% less leakage","How a dual-pronged framework made Chinese medical LLM SOTA and safer","Data framework lifts medical LLM to top ranks while cutting leaks 27%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1319,"prompt_tokens":750,"completion_tokens":569,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":461}},"tokens_in":494,"tokens_out":569,"duration_ms":5881,"temperature":1.0,"reasoning_tokens":461,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:36:20.259739+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the four datasets that appear in both the training corpus and the evaluation (CMtMedQA, webMedQA, CMExam, CMB-Exam), remove from the evaluation any question whose answer or phrasing appears in the training corpus (by exact match or high embedding similarity), and re-run the comparisons. If the accuracy margin against the open-source baselines largely vanishes after removing overlapped items, the SOTA claim is attributable to training-test leakage rather than to the data-processing pipeline.","supporting_citations":[],"review_version":1}