REVIEW 3 major objections 5 minor 25 references
A Comparative Study of Feature Selection Methods for EHR Diagnosis Codes in Opioid Use Disorder Prediction
T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper claims that NTK-motivated early gradient sensitivity is the best feature-selection method among five for OUD prediction from EHR diagnosis codes, with a plateau around 300 codes.
desk verdict A well-built benchmark with a novel NTK-based selector, but the head-to-head comparison is not clean: F11 codes leak into the data-driven feature sets while the LLM is explicitly told to exclude them. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the NTK-motivated early gradient sensitivity score. For each diagnosis $d$, the paper forms a recurrence-weighted patient feature $x'_d(p)=\log(1+f_d(p))$, initializes logistic-regression bias $b_0=\log(r/(1-r))$ from OUD prevalence $r$, and computes per-encounter gradient contributions $g_d(e)=(\hat p_0-y)\,x'_d(p)$. Codes are ranked by $S_d=\sum_e |g_d(e)|$. The NTK argument is that this initialization-time gradient magnitude forecasts how strongly each feature will influence the trained model, and the empirical claim is that this forecast is both accurate and far more stable under resampling than SHAP or Elastic Net attributions.
What would settle it
Retrain the NTK, LightGBM-SHAP, Elastic Net, and recurrence rankings after deleting every F11 and OUD-consequence code from the candidate vocabulary, then compare BERT AUPRC at $K=300$; if NTK's advantage shrinks or the selected subsets fall to LLM-only levels, the headline result is driven by label leakage.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a feature-selection method needing no iterative training—NTK-motivated early gradient sensitivity—outperforms recurrence enrichment, LightGBM-SHAP, Elastic Net, and LLM-guided semantic selection when the selected codes feed a BERT classifier. At $K=500$ it reaches test AUPRC $0.291$, AUROC $0.901$, and optimal F1 $0.319$, with near-deterministic rankings under bootstrap resampling (Jaccard similarity $0.984 \pm 0.006$ at $K=300$). The paper further claims that gains from larger budgets are concentrated below $K\approx 300$ and that consensus codes across methods retain usable signal, though the full 1,908-code vocabulary still yields higher AUPRC ($0.458$).
Load-bearing premise
The evaluation assumes the 12-month lookback features are temporally separate from the outcome, but OUD is defined as any F11 code in the patient's records and the paper never states that F11 or consequence codes were removed from the input vocabulary for the four data-driven methods.
Editorial extensions
If this is right
- A fixed vocabulary of roughly 300 three-character ICD-10 categories can match the predictive performance of much larger diagnosis-code sets, so deployment can use compact feature budgets without major loss.
- NTK sensitivity rankings are nearly deterministic under bootstrap resampling, so feature-selection results should reproduce across training-data perturbations.
- LLM-selected codes overlap little with data-driven rankings, so semantic priors can be used to expand candidate sets rather than replace statistical selection.
- Consensus feature sets selected by at least three methods perform between individual methods and the full vocabulary, suggesting that agreement across paradigms marks reliable signal.
- BERT-based downstream models outperform logistic regression and MLP under the same feature subsets, so architecture captures temporal diagnosis patterns beyond linear or shallow nonlinear models.
Reading between the lines
- If F11 codes were not removed from the data-driven vocabularies, the NTK advantage may shrink or vanish once exclusion is enforced; the paper's LLM prompt explicitly excludes them, so a fair comparison requires the same exclusion.
- If the roughly 300-code plateau holds across outcomes and institutions, health systems could standardize on a small, interpretable diagnosis vocabulary for OUD screening, improving validation and transportability.
- A hybrid pipeline that uses the LLM list to add rare clinically meaningful codes to the NTK list could exceed either method alone, but this paper does not test that combination.
- The gap between the selected subsets and the full 1,908-code vocabulary suggests substantial unused signal remains, so larger budgets or hierarchy-aware grouping could capture it.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports a benchmark of five feature-selection methods—recurrence enrichment, NTK-motivated early gradient sensitivity, LightGBM-SHAP, Elastic Net, and LLM-guided semantic selection—for selecting ICD-10 diagnosis categories to predict opioid use disorder (OUD) from EHR data. Using a Cerner Health Facts cohort of 11.8 million patients, the authors construct diagnosis features from the 12 months preceding an index opioid prescription, rank features by each method, and evaluate top-K subsets by training a BERT classifier from scratch. The reported findings are that NTK sensitivity gives the best accuracy/stability trade-off, that performance improves up to roughly 300 features and then plateaus, and that LLM selection adds complementary but weaker standalone signal. The paper is a comparative empirical study rather than a novel methodological contribution.
Significance. If the comparison were valid, it would provide practical guidance for diagnosis-code feature selection in sparse EHR spaces. Strengths of the manuscript include a very large real-world cohort, a unified preprocessing and evaluation pipeline, multiple feature budgets, downstream evaluation with a consistent classifier, bootstrap stability analysis, and the inclusion of statistical, ML, and LLM-based selectors. The NTK-motivated ranking is a simple and interesting baseline. However, the central comparison is compromised by an asymmetric label-leakage setup: the outcome-defining F11 code is apparently available to the four data-driven selectors but explicitly excluded from the LLM prompt. In addition, the full-vocabulary result contradicts the diminishing-returns claim. Because the headline conclusion rests on this comparison, the manuscript in its current form does not support its main claims.
major comments (3)
- [Methods, Data Source and LLM Semantic prompt; Table 1] The comparison is asymmetric because the outcome-defining code F11 is not excluded from the candidate vocabulary for the four data-driven selectors, while the LLM prompt explicitly excludes 'consequences or treatment of established OUD (i.e., label leakage)'. OUD is operationalized as any F11-prefixed diagnosis in the patient's records (Methods, Data Source), and the feature window is the 12 months preceding the index opioid prescription. If an F11 code appears in that window, the patient is OUD-positive by definition, so Recurrence Enrichment (Eq. 3), NTK Sensitivity (Eqs. 5-6), LightGBM-SHAP, and Elastic Net can all rank this feature as essentially a copy of the label. The paper never states that F11 was removed from the input vocabulary for these methods. This inflates their downstream AUPRC and stability and invalidates the headline claim that NTK sensitivity is the best selector. The limitation section calling for an incident-only sensitivity analysis is exactly the missing control; it should be a primary analysis, not a deferred caveat.
- [Overall Predictive Performance, Table 1] The 'diminishing returns around 300 features' conclusion is not supported by the reported numbers. Table 1 shows NTK Sensitivity at K=300 with AUPRC 0.247 and at K=500 with AUPRC 0.291, while the Full Vocabulary (1,908 features) reaches AUPRC 0.458, more than 50% higher than the best K=500 subset. This is not a marginal gain and is inconsistent with the abstract's claim that 'predictive performance improves rapidly up to a moderate vocabulary size (approximately 300 diagnosis categories) and then exhibits diminishing returns.' The full-vocabulary result is acknowledged in the text but only as a caveat; it should be reconciled with the conclusion, or the conclusion should be restricted to the evaluated range K=50-500.
- [Methods, Data Source and Downstream Model] The outcome definition is temporally ambiguous. The Data Source says patients are considered to have OUD if an F11 code appears 'in their diagnosis records,' with no statement that the code must occur after the feature window or after the index opioid prescription. The BERT model is instead described as predicting whether the patient 'will receive an OUD-related diagnosis at the next encounter.' If the input sequence already contains an F11 code, the model is being asked to predict a condition that is already documented. The paper should define an explicit index date, exclude F11 and related consequence/treatment codes from the feature window for all methods, and report an incident-only analysis; this is a prerequisite for the 'future OUD prediction' interpretation used throughout the abstract and conclusion.
minor comments (5)
- [Downstream Model] The BERT architecture and training hyperparameters (number of layers, hidden size, learning rate, batch size, training steps) are not specified; the statement that they are 'identical' across conditions is not sufficient for reproducibility.
- [Methods, Elastic Net] The grid-search ranges for the Elastic Net hyperparameters α and λ are not reported; include them so the selection procedure is reproducible.
- [Experiments and Results] The method is called 'LLM Semantic' in Methods but 'LLM Semantic Similarity' in the Experiments section; please use one name consistently.
- [Figure 2] There is a typo in the prompt text: 'Donotbase' should be 'Do not base'.
- [References] Reference [22] cites DrugBank for the ICD-9 to ICD-10 mapping, which does not appear to be the correct source for terminology mapping; please cite the appropriate mapping resource.
Circularity Check
Central OUD prediction is partially circular: F11 outcome codes are not excluded from the data-driven feature vocabulary, so the NTK advantage may reflect label leakage rather than genuine prediction.
-
self definitional
[Methods, Data Source; Methods, LLM Semantic prompt; Discussion, Limitations]
"Opioid use disorder (OUD) was operationalized using the ICD-10-CM category F11.xx. Patients were considered to have evidence of OUD if an F11-prefixed diagnosis appeared in their diagnosis records. ... For feature construction, diagnosis records were restricted to encounters occurring within the 12 months preceding the index opioid prescription. ... The model was instructed to assess clinical plausibility as an OUD precursor and to explicitly exclude codes representing consequences or treatments of established OUD to prevent label leakage."
The outcome is defined by the presence of F11 codes, and the candidate feature vocabulary for Recurrence Enrichment, NTK Sensitivity, LightGBM-SHAP, and Elastic Net is drawn from all diagnosis codes in the 12-month lookback window, with no stated exclusion of F11. A patient with an F11 code in that window is OUD-positive by definition, so the code is a perfect label. The reported 'predictions' therefore reduce, for patients carrying the code, to reading the outcome off the input. The LLM selector is explicitly told to exclude such codes to prevent label leakage, but the data-driven selectors are not, so the comparison is asymmetric and the NTK-best conclusion is partly an artifact of how strongly each method exploits the leaked code.
full rationale
The paper's evaluation is otherwise well-structured: feature selection uses only training/validation labels, metrics are computed on a held-out test set, and the downstream models are trained from scratch per method and budget. The authors' earlier opioid-prediction papers are cited as background and are not load-bearing for the new comparison; the NTK, SHAP, and Elastic Net methods are standard supervised procedures that are not circular by themselves. The one serious issue is the self-definitional label leakage described above: because OUD is operationalized by F11 codes and F11 codes are not removed from the data-driven feature vocabulary, the central predictive claim is partially equivalent to its input by construction. The paper's own limitation paragraph calls for 'incident-only definitions' and acknowledges 'temporal misalignment,' which is exactly the missing control. I therefore assign a partial-circularity score of 6: the NTK ranking itself is not fitted to the test result, but the prediction task it feeds is contaminated by the outcome code, so the headline result is not fully independent evidence.
Assumptions & free parameters
free parameters (4)
- Recurrence support threshold n_d >= 50 =
50
- BERT hyperparameters (architecture, training steps)
- LLM top-100 cap =
100
- Elastic Net alpha and lambda =
selected by grid search on validation AUROC
assumptions (5)
- domain assumption ICD-9 to ICD-10 mapping is correct and harmonizes the diagnosis space
- domain assumption Three-character ICD category truncation preserves the predictive signal
- domain assumption A BERT transformer can be trained from scratch on tokenized diagnosis sequences under the described protocol
- domain assumption The 12-month lookback window does not include the outcome code F11; no temporal leak
- ad hoc to paper NTK theory justifies ranking features by initial gradient magnitude
Cite this review
Pith. "Pith review of A Comparative Study of Feature Selection Methods for EHR Diagnosis Codes in Opioid Use Disorder Prediction." pith.science (2026). https://pith.science/paper/4RYQMNZ4
@misc{pith2026260804180,
author = {Pith},
title = {Pith review of: A Comparative Study of Feature Selection Methods for EHR Diagnosis Codes in Opioid Use Disorder Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/4RYQMNZ4}},
note = {Machine review of arXiv:2608.04180}
}
read the original abstract
Feature selection is a critical step in electronic health record (EHR)-based predictive modeling, where input variables are often high-dimensional, sparse, noisy, and redundant. Large feature sets not only increase computational burden and overfitting risk, but also make model interpretation difficult, leading to limited usefulness in clinical settings. In this study, we focus on diagnosis-related features and compare five feature selection paradigms for opioid use disorder (OUD) prediction: recurrence enrichment, NTK-motivated early gradient sensitivity, LightGBM-SHAP, Elastic Net, and large language model (LLM)-guided semantic selection. We use a unified preprocessing and evaluation framework and assess each method by downstream predictive performance, resampling stability, and representation of infrequent diagnosis codes. Our results demonstrate that performance improves with larger feature budgets with diminishing returns beyond a moderate size. NTK sensitivity provides the best overall balance of accuracy and stability, and LLM-guided selection contributes complementary clinically meaningful signals despite lower standalone performance.
Figures
Reference graph
Works this paper leans on
-
[1]
Abuse S, Administration MHS. Key Substance Use and Mental Health Indicators in the United States: Results from the 2019 National Survey on Drug Use and Health. Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration; 2020. HHS Publication No. PEP20-07- 01-001, NSDUH Series H-55. Availab...
work page 2019
-
[2]
Training: Assessing and Addressing Opioid Use Disorder; 2025
CDC. Training: Assessing and Addressing Opioid Use Disorder; 2025. Available from: https://www.cdc.gov/ overdose-prevention/hcp/trainings/assessing-and-addressing-opioid-use-disorder-oud.html
work page 2025
-
[3]
Rasmy L, Tiryaki F, Zhou Y , Xiang Y , Tao C, Xu H, et al. Representation of EHR data for predictive modeling: a comparison between UMLS and other terminologies. Journal of the American Medical Informatics Association. 2020;27(10):1593-9
work page 2020
-
[4]
Machine learning based opioid overdose prediction using electronic health records
Dong X, Rashidian S, Wang Y , Hajagos J, Zhao X, Rosenthal RN, et al. Machine learning based opioid overdose prediction using electronic health records. In: AMIA annual symposium proceedings. vol. 2019; 2020. p. 389
work page 2019
-
[5]
Dong X, Deng J, Hou W, Rashidian S, Rosenthal RN, Saltz M, et al. Predicting opioid overdose risk of patients with opioid prescriptions using electronic health records based on temporal deep learning. Journal of biomedical informatics. 2021;116:103725
work page 2021
-
[6]
An integrated LSTM-HeteroRGNN model for interpretable opioid overdose risk prediction
Dong X, Wong R, Lyu W, Abell-Hart K, Deng J, Liu Y , et al. An integrated LSTM-HeteroRGNN model for interpretable opioid overdose risk prediction. Artificial intelligence in medicine. 2023;135:102439
work page 2023
-
[7]
HIBERT: A Hybrid Clustering BERT for Interpretable Opioid Overdose Risk Prediction
Ding Z, Dong X, Liu Y , Ma T, Zhao X, Wong R, et al. HIBERT: A Hybrid Clustering BERT for Interpretable Opioid Overdose Risk Prediction. In: AMIA Annual Symposium Proceedings. vol. 2024; 2025. p. 303
work page 2024
-
[8]
Big data and machine learning in health care
Beam AL, Kohane IS. Big data and machine learning in health care. Jama. 2018;319(13):1317-8
work page 2018
Show all 25 references
-
[9]
Scalable and accurate deep learning with electronic health records
Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, et al. Scalable and accurate deep learning with electronic health records. NPJ digital medicine. 2018;1(1):18
2018
-
[10]
Doctor ai: Predicting clinical events via recurrent neural networks
Choi E, Bahadori MT, Schuetz A, Stewart WF, Sun J. Doctor ai: Predicting clinical events via recurrent neural networks. In: Machine learning for healthcare conference. PMLR; 2016. p. 301-18
2016
-
[11]
An introduction to variable and feature selection
Guyon I, Elisseeff A. An introduction to variable and feature selection. Journal of machine learning research. 2003;3(Mar):1157-82
2003
-
[12]
Regression shrinkage and selection via the lasso
Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology. 1996;58(1):267-88
1996
-
[13]
Deep patient: an unsupervised representation to predict the future of patients from the electronic health records
Miotto R, Li L, Kidd BA, Dudley JT. Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Scientific reports. 2016;6(1):26094
2016
-
[14]
On the stability of feature selection algorithms
Nogueira S, Sechidis K, Brown G. On the stability of feature selection algorithms. Journal of Machine Learning Research. 2018;18(174):1-54
2018
-
[15]
Clinical knowledge extraction via sparse em- bedding regression (KESER) with multi-center large scale electronic health record data
Hong C, Rush E, Liu M, Zhou D, Sun J, Sonabend A, et al. Clinical knowledge extraction via sparse em- bedding regression (KESER) with multi-center large scale electronic health record data. NPJ digital medicine. 2021;4(1):151
2021
-
[16]
Unsupervised feature selection to identify important ICD-10 and ATC codes for machine learning on a cohort of patients with coronary heart disease: retrospective study
Ghasemi P, Lee J. Unsupervised feature selection to identify important ICD-10 and ATC codes for machine learning on a cohort of patients with coronary heart disease: retrospective study. JMIR Medical Informatics. 2024;12(1):e52896
2024
-
[17]
Large language models facilitate the generation of electronic health record phenotyping algorithms
Yan C, Ong HH, Grabowska ME, Krantz MS, Su WC, Dickson AL, et al. Large language models facilitate the generation of electronic health record phenotyping algorithms. Journal of the American Medical Informatics Association. 2024;31(9):1994-2001
2024
-
[18]
Llm-select: Feature selection with large language models
Jeong DP, Lipton ZC, Ravikumar P. Llm-select: Feature selection with large language models. arXiv preprint arXiv:240702694. 2024
2024
-
[19]
A unified approach to interpreting model predictions
Lundberg SM, Lee SI. A unified approach to interpreting model predictions. Advances in neural information processing systems. 2017;30
2017
-
[20]
Cerner Health Facts ® Data Sets - SBMI Data Service; 2018
Cerner, of Texas Health Science Center at Houston (UTHealth) School of Biomedical Informatics TU. Cerner Health Facts ® Data Sets - SBMI Data Service; 2018. Accessed: November 25, 2018. Available from: https: //sbmi.uth.edu/sbmi-data-service/data-set/cerner/
2018
-
[21]
A comparison of a multistate inpatient EHR database to the HCUP Nationwide Inpatient Sample
DeShazo JP, Hoffman MA. A comparison of a multistate inpatient EHR database to the HCUP Nationwide Inpatient Sample. BMC health services research. 2015;15:1-8
2015
-
[22]
DrugBank 5.0: a major update to the DrugBank database for 2018
Wishart DS, Feunang YD, Guo AC, Lo EJ, Marcu A, Grant JR, et al. DrugBank 5.0: a major update to the DrugBank database for 2018. Nucleic Acids Research. 2017 11;46(D1):D1074-82. Available from: https://doi. org/10.1093/nar/gkx1037
2018 doi
-
[23]
Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis
Shickel B, Tighe PJ, Bihorac A, Rashidi P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE journal of biomedical and health informatics. 2017;22(5):1589- 604
2017
-
[24]
Case Study: Exploring How Opioid-Related Diagnosis Codes Translate From ICD-9- CM to ICD-10-CM
Moore BJ, Barrett ML. Case Study: Exploring How Opioid-Related Diagnosis Codes Translate From ICD-9- CM to ICD-10-CM. U.S. Agency for Healthcare Research and Quality; 2017. ONLINE. Accessed: 2025-03-04. Available from: https://www.hcupus.ahrq.gov/datainnovations/icd10 resources.jsp
2017
-
[25]
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin J, Chang MW, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume ...
2019
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.