REVIEW 3 major objections 4 minor 3 cited by
A data-processing pipeline lifts a 7B Chinese medical LLM to the top of open-source comparisons while cutting training-data leakage by 27%.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A data-processing and privacy-preserving deployment framework claims state-of-the-art Chinese medical LLM accuracy and a 27% reduction in training-data leakage.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection Likely train/eval overlap in the pre-training corpus undercuts the accuracy and SOTA claims; the engineering itself is plausible. the 3 major comments →
DPF-CM: A Data Processing Framework with Privacy-Preserving Vector Databases for Chinese Medical LLMs Training and Deployment
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper's central claim is that a comprehensive data lifecycle—cleaning and generating continued-pretraining data, generating question-oriented instructions via a chained-example prompt, selecting and optimizing SFT data, and denoising preference data with an ensemble of reward models—yields a Chinese medical LLM that achieves state-of-the-art accuracy among same-size open-source Chinese medical models on medical dialogue, medical benchmarks, and medical terminology explanation tasks. A second claim is that the PPVD deployment module, built from a high-risk embedding database and a matched secure database, can intercept and replace memorized training responses during inference, cutting mea
What carries the argument
The key mechanism on the training side is the chained-example instruction generation: seed question-instruction pairs are scored, sorted from low to high quality, and linked in a single prompt so the LLM learns a progressive refinement process and produces better instructions for new questions. The preference-data denoiser averages the 'preference distance' (reward-model score difference between chosen and rejected responses) across five independently trained reward models and drops samples whose distances are consistently too low or too high. On the deployment side, PPVD works by splitting training samples, using the first half as a prompt to elicit the model's memory, measuring output simi
Load-bearing premise
The claimed accuracy gains assume the evaluation benchmarks were not also part of the training data; the paper lists CMtMedQA, webMedQA, CMExam, and CMB-Exam as continued pre-training data and does not state that test splits were excluded.
What would settle it
Take the four datasets that appear in both the training corpus and the evaluation (CMtMedQA, webMedQA, CMExam, CMB-Exam), remove from the evaluation any question whose answer or phrasing appears in the training corpus (by exact match or high embedding similarity), and re-run the comparisons. If the accuracy margin against the open-source baselines largely vanishes after removing overlapped items, the SOTA claim is attributable to training-test leakage rather than to the data-processing pipeline.
If this is right
- Data processing alone, without architectural changes, can move a 7B medical model to the top of open-source Chinese medical comparisons.
- Chained-example instruction generation improves supervised fine-tuning more than generic few-shot prompting, especially for multi-turn medical dialogues.
- Averaging multiple reward models and removing extreme preference distances improves downstream DPO model accuracy.
- PPVD can reduce measured leakage of memorized training text from roughly 0.85 to 0.58 similarity while leaving ordinary response quality largely intact.
- The framework transfers to a different base model (Ziya-LLaMA-13B-v1), suggesting the processing gains are not specific to the Qwen base.
Where Pith is reading between the lines
- If the evaluation benchmarks also appear in the continued-pretraining corpus, as Table 16 suggests for CMtMedQA, webMedQA, CMExam, and CMB-Exam, part of the reported SOTA margin might reflect memorization of test items rather than genuine processing gains; a clean held-out evaluation would settle this.
- The 27% leakage reduction is measured only against a ROUGE-L similarity attack at a fixed threshold; other forms of attack such as paraphrase completion or membership inference could give different numbers.
- PPVD's dependence on storing full embeddings for high-risk samples could be a scalability bottleneck for very large private corpora; weight-agnostic sketches or locality-sensitive hashes might achieve similar protection with less storage, as the paper itself flags as a limitation.
- The chained-example instruction generation idea is a general data-augmentation technique that could be applied to other low-resource specialized domains beyond medicine, such as legal or financial question answering.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DPF-CM, a data processing framework for training and deploying Chinese medical LLMs, consisting of a training-data pipeline (continued pre-training, SFT, and preference-data denoising) and a deployment-time privacy-preserving vector database (PPVD) that identifies memorized training samples and replaces them with safe responses. The authors claim that DPF-CM significantly improves accuracy over an unprocessed baseline and achieves state-of-the-art results among open-source Chinese medical LLMs, while reducing training-data privacy leakage by 27%. The paper includes detailed data statistics, prompts, ablation studies, and comparisons with several baselines.
Significance. If validated, DPF-CM would be a useful holistic, data-centric contribution to Chinese medical LLM development, and its PPVD mechanism is a concrete engineering approach to mitigating training-data leakage at inference time. The paper is commendable for shipping a complete pipeline with reproducible prompts, data statistics, and extensive ablations. However, the central accuracy and SOTA claims rest on the assumption that evaluation data is disjoint from the training corpus, which is not demonstrated, and the privacy-leakage reduction is evaluated circularly. These are load-bearing validity concerns that must be resolved before the results can be accepted.
major comments (3)
- [§5.1 / Appendix D.1 / Table 16] The continued pre-training corpus in Table 16 includes CMtMedQA, webMedQA, CMExam, and CMB-Exam, while Section 6 evaluates on CMtMedQA, webMedQA, CMExam, and CMB. The text only states for webMedQA that "We use the test data to evaluate"; for CMtMedQA it mentions a 1000-item test set but does not say these items were excluded from the 2.5M-token pre-training corpus, and for CMExam/CMB-Exam no test split is mentioned. If any evaluation questions appear in the training data, the accuracy gains in Tables 4 and 9 and the win rates in Tables 5-6 could reflect memorization rather than genuine improvement. This directly affects the abstract's "significantly improves" and "state-of-the-art" claims. The authors must demonstrate disjointness (e.g., exact and fuzzy deduplication of evaluation items against the training corpus) or re-evaluate on a truly held-out benchmark.
- [§7.3] The claimed 27% privacy-leakage reduction is computed on the same 1,812 high-risk samples that were selected by the model-memory search and then protected. The protocol identifies samples with ROUGE-L > 0.85, applies the secure database, and re-measures average similarity on those same 1,812 samples. This is self-referential: it does not measure leakage on the other 98,181 samples or on non-flagged training samples, and it does not test adversarial prompts that may trigger memorization without being in the high-risk database. The privacy claim is therefore not established as a general property. Please evaluate on a held-out random sample of training data and on adversarial extraction or membership-inference prompts.
- [§6 / Tables 1-4, 9-12] No error bars, confidence intervals, or significance tests are reported. The paper repeatedly uses "significantly" (abstract and Section 6.1), but many benchmark differences are within one or two accuracy points (e.g., Table 4: CMMLU 0.79 vs 0.78; Table 9: PLE 0.69 for both Ours and GPT-4). Without variance or statistical testing, the significance claim is unsupported. Please report multiple seeds with standard deviations or bootstrap confidence intervals for the main comparisons.
minor comments (4)
- [§3.1] The first sentence of Section 3 refers to "PF-CMLT" rather than "DPF-CM"; please correct.
- [Table 5] The CMtMedQA row appears twice with different numbers in the second and third columns; the table needs to be deduplicated and corrected.
- [Appendix D.1] Only webMedQA is explicitly described as using the test split. Please state the exact evaluation split (train/test/exclusion) for every dataset used in Section 6.
- [Limitations] The limitations section does not mention the potential overlap between training and evaluation data. Please add a discussion of how the benchmarks were kept disjoint from the training corpus.
Circularity Check
Privacy-leakage reduction is measured on the same high-risk samples used to define the protection; the 27% drop is by construction.
specific steps
-
self definitional
[Section 7.3, 'Results of Data Privacy Protection' (with Section 4 'model memory searches')]
"We set the similarity threshold at 0.85. Through experimentation, we find that only 1,812 samples out of the 100,000 met the similarity criterion of greater than 0.85 ... Next, we execute the remaining steps and then retest the previously identified high-risk samples of 1,812. The experimental results showed that the average similarity of these samples decreased to 0.58, which is a reduction of approximately 0.27."
The 'high-risk' samples are defined in Section 4 as exactly those whose model output matches the training label with similarity above a threshold (ROUGE-L > 0.85). The protection mechanism then replaces the model response with a general-LLM response for those samples. The reported 27% privacy-leakage reduction is measured by re-computing similarity to the original labels on that same selected subset. Because the selected samples were chosen for high similarity to those labels, and because the replacement response is a different model's output, a large drop in similarity to the original label is essentially guaranteed by construction. The metric and the selection criterion are the same quantity; there is no independent attack set or held-out evaluation of privacy leakage. Thus the headline
full rationale
The training-pipeline claims (data cleaning, chained-example instruction generation, preference-data denoising, and ablations) are evaluated on standard medical benchmarks and do not reduce to their inputs. The preference-data denoising does train reward models on the same preference data they later filter, but that is a self-referential heuristic rather than a definitional equivalence; downstream accuracy is measured externally, so I do not count it as circular. The one genuine circularity is in the privacy evaluation: high-risk samples are defined by similarity to training labels, and the reported 27% reduction is the post-replacement similarity drop measured on exactly those selected samples. This makes the headline privacy number an artifact of the selection criterion. Separately, Table 16 lists CMExam and CMB-Exam as continued pre-training data while evaluation uses CMExam and CMB; this is a potential contamination/leakage risk for the accuracy claims, but it is a correctness risk rather than a circularity of derivation, so it does not increase the circularity score beyond the self-referential privacy metric.
Axiom & Free-Parameter Ledger
free parameters (6)
- High-risk memory similarity threshold =
0.85
- Match-and-replace cosine similarity threshold =
0.8
- Minhash-LSH deduplication threshold =
0.8
- Preference data score-cut percentiles =
top/bottom 10% removed
- SFT data selection score threshold =
9
- Annotation sample sizes =
4000 SFT + 2000 supplementary
axioms (3)
- domain assumption Evaluation benchmarks do not overlap with training corpora
- domain assumption GPT-4 is a reliable judge of medical response quality
- domain assumption ROUGE-L similarity above threshold indicates memorization
Cite this review
Pith. "Pith review of DPF-CM: A Data Processing Framework with Privacy-Preserving Vector Databases for Chinese Medical LLMs Training and Deployment." pith.science (2026). https://pith.science/paper/ESMR7T5X
@misc{pith2026250901354,
author = {Pith},
title = {Pith review of: DPF-CM: A Data Processing Framework with Privacy-Preserving Vector Databases for Chinese Medical LLMs Training and Deployment},
year = {2026},
howpublished = {\url{https://pith.science/paper/ESMR7T5X}},
note = {Machine review of arXiv:2509.01354}
}
read the original abstract
Current open-source training pipelines for Chinese medical language models predominantly emphasize optimizing training methodologies to enhance the performance of large language models (LLMs), yet lack comprehensive exploration into training data processing. To address this gap, we propose DPF-CM, a holistic Data Processing Framework for Chinese Medical LLMs training and deployment. DPF-CM comprises two core modules. The first module is a data processing pipeline tailored for model training. Beyond standard data processing operations, we (1) introduce a chained examples context-learning strategy to generate question-oriented instructions to mitigate the lack of instruction content, and (2) implement an ensemble-based filtering mechanism for preference data curation that averages multiple reward models to suppress noisy samples. The second module focuses on privacy preservation during model deployment. To prevent privacy risks from the inadvertent exposure of training data, we propose a Privacy Preserving Vector Database (PPVD) approach, which involves model memory search, high-risk database construction, secure database construction, and match-and-replace, four key stages to minimize privacy leakage during inference collectively. Experimental results show that DPF-CM significantly improves model accuracy, enabling our trained Chinese medical LLM to achieve state-of-the-art performance among open-source counterparts. Moreover, the framework reduces training data privacy leakage by 27%.
Figures
Forward citations
Cited by 3 Pith papers
-
MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agents
MemPrivacy replaces privacy-sensitive spans with structured placeholders on edge devices to enable effective cloud memory management while limiting utility loss to 1.6% and outperforming general models on privacy extraction.
-
MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agents
MemPrivacy uses edge detection of sensitive spans and type-aware placeholders to enable cloud-side memory management for LLM agents without exposing private data, achieving under 1.6% utility loss.
-
MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agents
MemPrivacy uses edge-side privacy span detection and semantic placeholders to enable cloud memory management for LLM agents while limiting utility loss to 1.6% and outperforming masking baselines.
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
Pith/arXiv arXiv 2023
-
[2]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609
Pith/arXiv arXiv 2023
-
[3]
Zhijie Bao, Wei Chen, Shengze Xiao, Kuang Ren, Jiaao Wu, Cheng Zhong, Jiajie Peng, Xuanjing Huang, and Zhongyu Wei. 2023. https://arxiv.org/abs/2308.14346 Disc-medllm: Bridging general large language models and real-world medical consultation . Preprint, arXiv:2308.14346
Pith/arXiv arXiv 2023
-
[4]
Junying Chen, Xidong Wang, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, Jianquan Li, Xiang Wan, Haizhou Li, and Benyou Wang. 2024. https://arxiv.org/abs/2311.09774 Huatuogpt-ii, one-stage training for medical adaption of llms . Preprint, arXiv:2311.09774
Pith/arXiv arXiv 2024
-
[5]
Junqing He, Mingming Fu, and Manshu Tu. 2019. https://doi.org/10.1186/s12911-019-0761-8 Applying deep matching networks to chinese medical question answering: a study and a dataset . BMC Medical Informatics and Decision Making, 19(2):52
-
[6]
Wei Huang, Yinggui Wang, Anda Cheng, Aihui Zhou, Chaofan Yu, and Lei Wang. 2024. A fast, performant, secure distributed training framework for llm. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4800--4804. IEEE
work page 2024
-
[7]
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, jiayi lei, Yao Fu, Maosong Sun, and Junxian He. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/c6ec1844bec96d6d32ae95ae694e23d8-Paper-Datasets_and_Benchmarks.pdf C-eval: A multi-level multi-discipline chinese evaluation...
work page 2023
-
[8]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611--626
2023
-
[9]
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. https://aclanthology.org/2024.findings-acl.671 CMMLU : Measuring massive multitask language understanding in C hinese . In Findings of the Association for Computational Linguistics ACL 2024, pages 11260--11285, Bangkok, Thailand and virtual meeting....
work page 2024
-
[10]
Yusheng Liao, Shuyang Jiang, Yu Wang, and Yanfeng Wang. 2024. Ming-moe: Enhancing medical multi-task learning in large language models with sparse mixture of low-rank adapter experts. arXiv preprint arXiv:2404.09027
Pith/arXiv arXiv 2024
-
[11]
Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, LEI ZHU, and Michael Lingzhi Li. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/a48ad12d588c597f4725a8b84af647b5-Paper-Datasets_and_Benchmarks.pdf Benchmarking large language models on cmexam - a comprehensive chinese medica...
work page 2023
-
[12]
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. https://doi.org/10.18653/v1/2022.acl-short.8 P -tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61--68, Dublin...
-
[13]
I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101
Pith/arXiv arXiv 2017
-
[14]
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE
2020
-
[15]
GLM Team, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv e-prints, pages arXiv--2406
work page 2024
-
[16]
Yuanhe Tian, Ruyi Gan, Yan Song, Jiaxing Zhang, and Yongdong Zhang. 2024. https://aclanthology.org/2024.acl-long.386 C hi M ed- GPT : A C hinese medical large language model with full training regime and better alignment to human preferences . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...
work page 2024
-
[17]
Xidong Wang, Guiming Chen, Song Dingjie, Zhang Zhiyi, Zhihong Chen, Qingying Xiao, Junying Chen, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, and Haizhou Li. 2024. https://doi.org/10.18653/v1/2024.naacl-long.343 CMB : A comprehensive medical benchmark in C hinese . In Proceedings of the 2024 Conference of the North American Chapter of the Association ...
-
[18]
Yinggui Wang, Wei Huang, and Le Yang. 2023. Privacy-preserving end-to-end spoken language understanding. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 5224--5232
work page 2023
-
[19]
Winning. 2023. https://github.com/winninghealth/WiNGPT2 Wingpt2
work page 2023
-
[20]
Honglin Xiong, Sheng Wang, Yitao Zhu, Zihao Zhao, Yuxiao Liu, Linlin Huang, Qian Wang, and Dinggang Shen. 2023. https://arxiv.org/abs/2304.01097 Doctorglm: Fine-tuning your chinese doctor is not a herculean task . Preprint, arXiv:2304.01097
Pith/arXiv arXiv 2023
-
[21]
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305
Pith/arXiv arXiv 2023
-
[22]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024 a . Qwen2 technical report. arXiv preprint arXiv:2407.10671
Pith/arXiv arXiv 2024
-
[23]
Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jia, and Hongying Zan. 2024 b . https://doi.org/10.1609/aaai.v38i17.29907 Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue . Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):1...
-
[24]
Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, et al. 2023. Huatuogpt, towards taming language model to be a doctor. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10859--10885
work page 2023
-
[25]
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223
Pith/arXiv arXiv 2023
-
[26]
Yanxin Zheng, Wensheng Gan, Zefeng Chen, Zhenlian Qi, Qian Liang, and Philip S Yu. 2025. Large language models for medicine: a survey. International Journal of Machine Learning and Cybernetics, 16(2):1015--1040
work page 2025
-
[27]
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372
Pith/arXiv arXiv 2024
-
[28]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[29]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.