{"id":"3f40fda6-6fd8-4ad4-9da5-21731202de7a","arxiv_id":"2506.15118","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CKD-EHR combines efficacy-aware data augmentation with LLM-to-BERT knowledge distillation and reports gains on the MIMIC-III 25-disease phenotyping task.","lead":"This paper describes a framework that trains a small BERT model for predicting which diseases a patient will have at their next hospital visit, using knowledge distilled from a large language model. The authors report improved accuracy over several published baselines on the MIMIC-III dataset, along with a large reduction in inference time and memory.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EADF's efficacy feature q is derived from the next-visit outcome d(i+1) and is fed as input; unless Q is training-fold-only, the reported MIMIC-III gains are contaminated by target leakage.","rationale":"I read the paper as claiming that CKD-EHR improves MIMIC-III prediction via EADF plus LoRCKD. The least secure part is EADF's q construction: it appears to use the outcome d(i+1) to select an input feature, and the paper gives no train/test separation for Q and no deployment-time definition of q. This is more load-bearing than the reader's baseline-split concern because even perfectly matched baselines would inherit the contamination. The ablation table is not conclusive, since it also lacks a true no-leakage control. The correct response is to reject the current version and require the leakage-controlled re-run before the central claim can be evaluated.","tokens_in":12244,"tokens_out":6443,"duration_ms":71665,"concrete_test":"Re-run the full pipeline on MIMIC-III in two leakage-controlled variants: (1) omit q from the augmented input entirely; (2) compute Q only on the training folds and use it for test visits. If CKD-EHR's ACC/F1 over Bio+ClinicalBERT collapses, or drops back to the no-EADF row of Table 3 (ACC 94.75, F1 79.56), the claimed gains come from target leakage rather than from distillation. Also report the same comparison with a non-leaking efficacy feature added to the baselines.","verdict_should_be":"REJECT","load_bearing_attack":"The central claimed gains rest on EADF. In Sec. 3.2, Q is built by pairing each visit with the next visit and 'analyzing changes in disease state across visit pairs'; q is then the top treatment for d(i) and is inserted into the input text. This means q encodes the future outcome d(i+1) unless Q is estimated strictly on training folds. The paper never states such a split, and Sec. 4.3.1 reports results against published baselines without describing how q is obtained for test visits. If Q uses the full MIMIC-III cohort, the test labels leak into the input feature, so the Table 1 improvements (ACC 94.76 vs. ClinicalBERT 94.62; F1 80.25 vs. 76.78) and the headline 9%/27% gains are not evidence for the method. Even a training-only Q requires a baseline that also receives q to isolate distillation benefits; none is provided.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CKD-EHR, a knowledge-distillation framework for EHR-based disease prediction. A Qwen2.5-7B teacher is fine-tuned with LoRA on data augmented by the Efficacy-Aware Data Fusion (EADF) module, which uses visit pairs from MIMIC-III to generate efficacy-ranked treatment information; a Multi-Label Adaptive Projection Head (MLAPH) converts the teacher's hidden states into soft labels, and a BERT student is trained on a weighted combination of soft and hard labels. Experiments on MIMIC-III for 25 disease-prediction tasks report accuracy 94.76, F1 80.25, AUC 91.11, AUPR 76.33, a 22.2x inference speedup, and a 34.96x memory reduction relative to the teacher model. The abstract and conclusion claim a 9% accuracy increase and a 27% F1 improvement over baseline models with p<0.01.","tokens_in":12470,"tokens_out":3477,"duration_ms":37072,"significance":"If the reported results are valid, the framework would provide a practical approach to compressing large language models for clinical prediction while retaining accuracy. The paper has notable strengths: the authors make code and data available, provide concrete efficiency measurements, include ablation studies for the main components, and explore alternative soft-label calculation methods. The central claim, however, rests on three load-bearing issues that currently undermine confidence: the headline improvement numbers do not match Table 1, the EADF feature construction may leak future-visit information into the input, and the baseline comparisons are not documented as sharing a common evaluation protocol. The manuscript is technically interesting and potentially useful, but these issues must be resolved before the performance claims can be accepted.","major_comments":[{"comment":"The abstract and conclusion state that CKD-EHR increases diagnostic accuracy by 9% and F1-score by 27% compared with the baseline. These numbers are not derivable from Table 1: the accuracy improvement over the strongest published baseline (ClinicalBERT) is 0.14 absolute points (about 0.15% relative), and the F1 improvement is 3.47 absolute points (about 4.5% relative). The paper does not specify which baseline the 9% and 27% refer to, nor whether these are relative or absolute changes. Please report the exact baseline used, the calculation method, and align the abstract and conclusion with the numbers in Table 1.","section":"Abstract, Section 4.3.1 (Table 1), Section 5"},{"comment":"The EADF module constructs q, a ranked treatment-efficacy list, by pairing each visit with the next visit and analyzing changes in disease state. The augmented input includes q, which is derived from d(i+1), the next-visit diagnosis. If Q is estimated on the full MIMIC-III cohort rather than only on training folds, then test-set labels leak into the test inputs, making the reported performance gains invalid. The manuscript never states that Q is computed strictly from training data. Even if Q is training-only, the comparison lacks a baseline that also receives q as an input feature, so the observed gains cannot be attributed specifically to knowledge distillation rather than to the additional feature. Please clarify the splitting procedure for Q and add a control baseline that receives the same augmented input without distillation.","section":"Section 3.2 (EADF) and Section 4.3.1"},{"comment":"The paper claims statistical significance (p<0.01) in the abstract, but Section 4.2 describes no statistical tests, no confidence intervals, no error bars, and no cross-validation or repeated-run variability. In addition, Table 1 appears to compare against published baseline numbers without stating whether all models were trained and evaluated on the same MIMIC-III train/validation/test split, the same set of 25 disease labels, and the same aggregation rule for multi-label metrics. Without a shared evaluation protocol and variance estimates, the comparison to baselines is not meaningful. Please provide per-fold or repeated-run results, statistical significance tests, and a description of the common data split and metric aggregation used for all models, including the baselines.","section":"Section 4.2 and Section 4.3.1"},{"comment":"The comprehensive ablation shows that the full CKD-EHR model (EADF + LoRCKD) achieves F1=80.25, whereas the teacher with EADF only achieves F1=82.94 and the teacher without either component achieves F1=81.61. Thus, the distillation step reduces F1 by 2.69 points relative to the teacher with EADF, and the only metric where the full model is competitive is accuracy (94.76 vs. 94.75). This pattern weakens the claim that LoRCKD improves prediction quality; the paper's own text acknowledges that F1, AUC, and AUPR are lower for the full model than for the teacher. The contribution of distillation should be reframed as an efficiency-accuracy trade-off, and the ablation discussion should quantify the cost of the F1 drop rather than presenting the result as a clear success.","section":"Section 4.4.1 (Table 3)"}],"minor_comments":[{"comment":"The phrase '10 iterations' should be clarified as 10 training epochs; as written it is ambiguous whether the student model is trained for 10 iterations over the whole dataset or 10 update steps.","section":"Section 4.2"},{"comment":"The baseline name 'GrphCare' appears to be a typo for 'GraphCare', which is spelled correctly in Section 4.2.1 and in the reference list.","section":"Table 1"},{"comment":"The text states that alpha and beta are weighting parameters with alpha + beta = 1, but the range of alpha tested in Section 4.5 is not formally stated. In addition, 'soft target contribution rates' in the caption of Fig. 2 should be defined as the beta value or as a complement of alpha.","section":"Section 3.3, Eq. (6)"},{"comment":"The confusion-matrix figure captions do not state whether the matrices are computed on the test set or on a validation set; please specify the data split used for visualization.","section":"Section 4.6"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the central claims are currently not supported by the numbers as reported, and the EADF construction raises a leakage concern that the authors need to address explicitly. If the study was conducted with training-fold-only efficacy statistics, the paper can likely be revised to a publishable state; if not, the headline results would be invalid. I recommend requiring the authors to clarify this point and to re-run the baseline comparison under a shared protocol before further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper describes a three-part pipeline: EADF augments EHR text with treatment-efficacy signals, a LoRA-fine-tuned Qwen2.5-7B teacher generates soft labels via MLAPH, and a BERT student distills them. The combination is new, and the authors are honest enough to run a full ablation. But the evidence does not support the headline claims, and the EADF design leaks the future.\n\nWhat is actually new: this is the first LLM-to-BERT distillation for EHR phenotyping that I know of, and using a linear projection head on teacher hidden states to emit soft labels is a sensible way to avoid vocabulary-based KL divergence. If the evaluation were clean, the 22x speedup over a 7B teacher would be a useful result for resource-constrained clinical settings.\n\nWhere it falls apart: EADF computes q, the top treatment per disease, from visit pairs that include d(i+1), the next-visit diagnosis. That is the label for the prediction task. The paper never says Q is estimated on training folds alone. If Q is computed on the full MIMIC-III cohort, then for test visits the input text contains information aggregated from their own future outcomes, and Table 1's small gains over ClinicalBERT (ACC 94.76 vs 94.62, F1 80.25 vs 76.78) are not evidence for the method. Even a training-only Q would not isolate the distillation benefit, because the student receives an augmented input feature that none of the baselines get.\n\nThe reporting has other problems. The abstract's 9% accuracy and 27% F1 numbers do not match Table 1 comparisons to any listed baseline. The p<0.01 in the abstract appears nowhere in the methods. The ablation shows the full model underperforms the teacher-with-EADF on F1, AUC, and AUPR, which undercuts the claim that distillation helps. And the baseline numbers are quoted without a common split or protocol.\n\nWho this is for: someone working on model compression for clinical NLP might borrow the MLAPH idea, but not the evaluation. The paper needs a redo with strict temporal splitting, a leakage-free augmentation, and proper baselines.\n\nIf I were the editor, I would not send this out in its current form. The central claim is contaminated by likely target leakage, and the significance claim is unverifiable. I would tell the authors to fix EADF and rerun.","headline":"Plausible pipeline, but EADF likely leaks the next-visit diagnosis into the input, so the reported gains cannot be trusted.","tokens_in":12928,"tokens_out":5335,"would_cite":false,"duration_ms":50910,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that clinical knowledge can be distilled from a 7-billion-parameter LLM into a compact BERT model, improving MIMIC-III disease prediction while cutting inference time by 22.2 times.","keywords":["clinical knowledge distillation","electronic health records","disease prediction","MIMIC-III","large language models","LoRA fine-tuning","soft labels","multi-label classification"],"falsifier":"Re-run all models in Table 1 from a single MIMIC-III phenotyping split with identical preprocessing, and check whether CKD-EHR's ACC, F1, AUC, and AUPR gaps over RAM-EHR and ClinicalBERT persist; if the F1 advantage over the best clinical BERT baseline drops below a few points, the central claim of large-model knowledge transfer is not supported.","tokens_in":12105,"feed_emoji":"🩺","tokens_out":8875,"duration_ms":82564,"temperature":0.7,"pith_summary":"The paper tries to establish that a large medical language model can be turned into a small, fast, and accurate disease-prediction model through a three-stage distillation framework called CKD-EHR. A Qwen2.5-7B teacher is first fine-tuned with LoRA on patient records enriched by Efficacy-Aware Data Fusion, which ranks medication and procedure combinations by their observed effect on the next visit's diagnosis. The teacher's hidden states are projected through a multi-label head into 25 disease probabilities that serve as soft labels, and a BERT student is trained on a weighted mix of these soft labels and the true labels. On MIMIC-III the student reaches 94.76% accuracy and 80.25 F1, roughly 9% higher accuracy and 27% higher F1 than the retrieval-augmented baseline it is compared against, while running 22.2 times faster and using about 35 times less memory than the 7B teacher. The broader point the authors want a reader to accept is that distilled soft knowledge plus real clinical treatment signals can make lightweight models practical for clinical decision support.","feed_headline":"CKD-EHR distills a 7B LLM's clinical knowledge into a fast BERT model","feed_subtitle":"CKD-EHR lifts MIMIC-III F1 to 80.25 while cutting inference time 22x and GPU memory ~35x.","key_machinery":"The load-bearing mechanism is the trio of EADF, LoRCKD, and MLAPH. EADF converts pairs of consecutive visits into natural-language records and attaches a rank q of the most effective treatment for the current diagnosis, computed statistically from the dataset, thereby injecting an efficacy signal. LoRCKD freezes Qwen2.5-7B's weights and trains only LoRA low-rank matrices in the attention modules, then extracts 25-dimensional soft-label probabilities via MLAPH, a linear projection of pooled hidden states. The BERT student minimizes BCEWithLogitsLoss on a weighted combination of hard and soft labels, so the knowledge transfer path runs from real outcomes to ranked efficacies to teacher hidden states to soft probabilities to student logits. MLAPH is what avoids the standard vocabulary-logit KL divergence, which would be expensive and noisy for generative models.","core_discovery":"The central discovery is that the distilled student can carry the teacher's clinical performance: CKD-EHR reaches ACC 94.76, F1 80.25, AUC 91.11, and AUPR 76.33 on the MIMIC-III 25-phenotype task, outperforming all listed baselines including ClinicalBERT, BioBERT, and Bio+ClinicalBERT. The largest gap is against RAM-EHR (ACC 85.54, F1 53.01), matching the abstract's stated 9% accuracy and 27% F1 improvements. The ablation shows that EADF alone gives the largest F1 lift (81.61 to 82.94 on the teacher), while adding LoRCKD trades a little F1 for large efficiency gains; the full student model, with hard-label weight α=0.9, sits at 80.25 F1. The authors interpret this as evidence that augmentation built from the target EHR's own visit outcomes beats retrieving generic text, and that soft labels produced from the teacher's hidden states transfer richer uncertainty and inter-label dependencies than hard labels alone.","pith_inferences":["Editorial inference: The headline gains are measured against published baseline numbers; if every baseline were retrained on the exact CKD-EHR split, the ranking could shift even if the method's own results stand.","Editorial inference: EADF's efficacy ranking is derived from observed visit transitions, so it is associational rather than causal; the pipeline would be a stronger clinical tool if the ranking were validated against randomized or adjusted treatment effects.","Editorial inference: The same teacher–projection-head–student recipe should transfer to other structured prediction tasks such as MIMIC-IV phenotyping or length-of-stay prediction; a cheap test is whether F1 gains persist when the teacher is a smaller instruction-tuned model.","Editorial inference: The paper itself limits the claim to one dataset and 25 phenotypes; extending to multimodal data or thousands of codes is not established and would face label-sparsity problems like the false negatives seen for low-support diseases."],"forward_implications":["If the results hold, a hospital can deploy BERT-scale models that deliver most of a 7B model's disease-prediction accuracy on standard hardware.","The 22.2x inference speedup and 34.96x memory reduction make real-time screening on large EHR populations feasible without large GPU clusters.","Building the augmentation knowledge from the dataset's own visit outcomes, rather than web text, gives distillation access to treatment-effect signals that RAG-style retrieval does not provide.","Soft labels produced in the fixed disease-label space via MLAPH let the student learn inter-label dependencies and uncertainty, which is why the full method beats hard-label-only training in the α=0.9 setting."],"supporting_citations":[{"why":"Supplies the MIMIC-III critical care database from which all patient visit pairs, diagnoses, medications, and procedures are drawn.","marker":"[25]"},{"why":"Defines the phenotyping benchmark of 25 binary disease prediction tasks that the paper uses for training and evaluation.","marker":"[26]"},{"why":"RAM-EHR is the strongest retrieval-augmented EHR baseline and the reference point for the reported 9% accuracy and 27% F1 gains.","marker":"[10]"},{"why":"ClinicalBERT is the best-performing clinical-text baseline in Table 1 that CKD-EHR must beat on accuracy and F1.","marker":"[35]"},{"why":"Bio+ClinicalBERT is another clinical BERT baseline included in the main comparison table.","marker":"[37]"},{"why":"BioBERT is a biomedical language model baseline used to demonstrate gains over domain-specific BERT models.","marker":"[36]"},{"why":"Cited for the LoRA-based fine-tuning and distillation procedure used by LoRCKD to adapt the teacher efficiently.","marker":"[11]"},{"why":"Supports the multi-label adaptive projection head (MLAPH) that maps teacher hidden states to disease-label probabilities.","marker":"[12]"},{"why":"Frames the retrieval-augmented generation background that EADF replaces by building knowledge from real EHR outcomes instead of web text.","marker":"[4]"}],"fun_headline_variants":["CKD-EHR: 22x faster inference, 27% better F1 via LLM distillation","Distill 7B Qwen2.5 into BERT: 22x speedup, 27% F1 boost","BERT student beats ClinicalBERT: distilled from 7B teacher with 22x speed","LLM knowledge distilled to BERT: 22x faster, 27% better F1 on MIMIC-III","CKD-EHR: 7B teacher to BERT student-22x faster and 27% F1 gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes every baseline in Table 1 was trained and evaluated on the same MIMIC-III train/test division under comparable settings, yet the paper (Sections 4.2.1 and 4.3.1) does not state that this equivalence was enforced.","fun_headline_variants_meta":{"raw":{"variants":["CKD-EHR: 22x faster inference, 27% better F1 via LLM distillation","Distill 7B Qwen2.5 into BERT: 22x speedup, 27% F1 boost","BERT student beats ClinicalBERT: distilled from 7B teacher with 22x speed","LLM knowledge distilled to BERT: 22x faster, 27% better F1 on MIMIC-III","CKD-EHR: 7B teacher to BERT student-22x faster and 27% F1 gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001133,"raw_usage":{"total_tokens":4742,"prompt_tokens":1013,"completion_tokens":3729,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":3591}},"tokens_in":629,"tokens_out":3729,"duration_ms":22364,"temperature":1.0,"reasoning_tokens":3591,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:42:18.940798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run all models in Table 1 from a single MIMIC-III phenotyping split with identical preprocessing, and check whether CKD-EHR's ACC, F1, AUC, and AUPR gaps over RAM-EHR and ClinicalBERT persist; if the F1 advantage over the best clinical BERT baseline drops below a few points, the central claim of large-model knowledge transfer is not supported.","supporting_citations":[{"cited_title":"Harutyunyan, H","cited_arxiv_id":null,"evidence_quote":"Defines the phenotyping benchmark of 25 binary disease prediction tasks that the paper uses for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BioBERT is a biomedical language model baseline used to demonstrate gains over domain-specific BERT models."},{"cited_title":"Ridnik, G","cited_arxiv_id":null,"evidence_quote":"Supports the multi-label adaptive projection head (MLAPH) that maps teacher hidden states to disease-label probabilities."},{"cited_title":"Lewis, E","cited_arxiv_id":null,"evidence_quote":"Frames the retrieval-augmented generation background that EADF replaces by building knowledge from real EHR outcomes instead of web text."}],"review_version":2}