REVIEW 4 major objections 5 minor 35 references
A Dynamic Framework for Semantic Grouping of Common Data Elements (CDE) Using Embeddings and Clustering
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Embedding CDE text and clustering with HDBSCAN yields 118 semantic clusters and 90.46% classification accuracy.
desk verdict Useful end-to-end pipeline for CDE grouping, but the headline accuracy is measured only on the 26% of CDEs that clustered, so the 'assign new/unclustered CDEs' claim is unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the dense vector embedding: each CDE's designation, definition, and permissible values are concatenated into a single string and passed through OpenAI's text-embedding-3-small model to produce a 1,536-dimensional vector in which semantically similar CDEs land close together. HDBSCAN (hierarchical density-based spatial clustering of applications with noise) then groups these vectors into clusters of arbitrary shape and density without requiring the number of clusters in advance, while labeling points that do not fit as outliers ($-1$). Cluster labels come from GPT-3.5-turbo summarization of up to 20 representative CDEs per cluster. A Random Forest classifier using the same embeddings as features learns to predict those cluster labels, turning the embedding space into an operational routing tool for new or previously unclustered CDEs.
What would settle it
Take the 17,973 outlier CDEs, embed them with the same model, and count how many lie closer to one of the 118 cluster centroids than the median within-cluster distance of that cluster. If a large fraction (for example, more than a quarter) are that close, the 'outliers are semantically unique' premise fails and the reported coverage overstates what the framework can group.
Extended reading notes
Core claim
The central claim is that semantic heterogeneity among Common Data Elements can be addressed by representation rather than by hand-built mapping rules. Concatenating each CDE's preferred designation, definition, and permissible values into one text string, embedding that string with OpenAI's text-embedding-3-small model, and clustering the resulting 1,536-dimensional vectors with HDBSCAN yields groups that correspond to recognizable biomedical and social concepts. With min_cluster_size set to 20, the framework found 118 clusters in the NIH NLM repository; a Random Forest trained on the embeddings assigned held-out CDEs to the correct LLM-generated cluster label with 90.46% overall accuracy; and the same method applied to Gravity Projects SDOH data produced clusters that matched the 21 known domains at ARI 0.52 and NMI 0.78. The framework is offered as a scaffold for harmonization—suggesting which CDEs belong together—rather than as a replacement for formal semantic mapping.
Load-bearing premise
The load-bearing premise is that the roughly three-quarters of repository CDEs that HDBSCAN labeled as outliers are genuinely semantically unique or unimportant, so that the 118 clusters and the 90.46% accuracy—both computed on the clustered subset—describe the framework's real coverage of the full repository.
Editorial extensions
If this is right
- New CDEs entering a repository can be assigned to an existing labeled cluster automatically, so curators see a suggested grouping immediately rather than starting from scratch.
- CDEs from different steward organizations (LOINC, NCI, NHLBI, GRDR, and others) land in the same cluster when they mean the same thing, which the 'Educational Attainment Levels' example illustrates.
- The external validation against SDOH domains indicates the same embedding-plus-HDBSCAN recipe transfers to at least one other CDE collection with known domain labels.
- For well-populated semantic categories the classifier is reliable, but the paper's own class-level report shows that small or sparse clusters can collapse to zero precision and recall.
- The 118 clusters provide a concrete shortlist of candidate CDEs for harmonization, filling a gap the paper identifies in earlier repository-alignment work.
Reading between the lines
- Beyond the paper: a stricter test would compare the LLM-generated cluster labels against a human-curated medical ontology to see whether the 118 clusters correspond to recognized concepts, not just to coherent text clusters.
- Beyond the paper: the 73.8% outlier rate suggests a two-pass design—cluster first, then re-embed the outliers with a smaller minimum cluster size or assign them to the nearest existing cluster by a cosine-similarity threshold—to recover rare but meaningful subconcepts the single pass discards.
- Beyond the paper: if the same embedding space were applied across multiple CDE repositories at once, it could expose cross-repository synonyms that the paper only demonstrates within one repository plus a single external benchmark.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an end-to-end framework for harmonizing Common Data Elements (CDEs): (1) generate LLM embeddings (OpenAI text-embedding-3-small) of CDE text, (2) cluster the embeddings with HDBSCAN, (3) assign cluster labels via GPT-3.5-turbo summarization, and (4) train a Random Forest classifier to assign new or unclustered CDEs to these labels. The framework is evaluated on the NIH NLM CDE Repository (24,363 entries) and externally on the Gravity Projects SDOH dataset. The authors report 118 clusters at min_cluster_size=20, a classification accuracy of 90.46%, and external validation with ARI 0.52 and NMI 0.78. The conclusion acknowledges that 73.8% of NIH entries and 54.1% of Gravity entries were labeled outliers by HDBSCAN.
Significance. If the reported results were valid as stated, the framework would offer a practical, scalable tool for suggesting harmonization candidates across CDE repositories, with a clear pipeline and reproducible supplementary code. The use of HDBSCAN to handle noise and the inclusion of external validation data are commendable. However, several evaluation gaps directly affect the central claims: the accuracy figure applies only to the 26.2% of entries that formed clusters, the classification evaluation is largely circular because the same embeddings generate both the cluster labels and the classifier features, and the Gravity validation parameter was selected after inspecting the external metrics. These issues currently prevent the conclusions from being supported at the level claimed.
major comments (4)
- [Abstract; Classification Evaluation Using LLM-Derived Labels from NIH CDEs; Conclusion] The 90.46% accuracy is computed only on entries assigned to the 118 HDBSCAN clusters, i.e., 6,390 of 24,363 (26.2%), while 17,973 entries (73.8%) are outliers. The abstract and conclusion state an 'overall accuracy' without this restriction, and the classifier is described as a mechanism to assign 'new or unclustered CDEs' to labeled clusters. No experiment evaluates classification of the outlier population, either as a separate class or via forced assignment. Because the outlier population is the majority of the repository, the reported accuracy does not support the framework's stated utility for the full data distribution. Please report metrics on the full dataset (including an outlier/reject option) or clearly qualify the accuracy as applying only to the non-outlier subset, and discuss how the intended use case handles the 73.8% outliers.
- [Classification; Classification Evaluation Using LLM-Derived Labels from NIH CDEs] The classification evaluation is substantially circular. HDBSCAN assigns cluster labels from the embeddings; the Random Forest is then trained to predict those same cluster labels from the same embeddings. High test accuracy therefore mainly demonstrates that the clusters are separable in the embedding space with respect to the labels HDBSCAN produced, not that the clusters correspond to semantically meaningful harmonization groups or that a new/unclustered CDE would be routed to a valid group. An independent ground truth is needed. As a concrete remedy, evaluate classification on the Gravity SDOH dataset using the known SDOH domain labels as targets, and/or provide a human-annotated gold standard for a sample of NIH CDEs. Without such an evaluation, the '90.46% accuracy' claim is not evidence of semantic grouping quality.
- [Evaluation; Table 7] The choice of min_cluster_size=7 for the Gravity dataset appears to have been made after inspecting the external ARI/NMI values across candidate sizes: min_cluster_size=7 gives ARI 0.5239 and NMI 0.7768, while min_cluster_size=10 gives ARI 0.6176 and NMI 0.8009. Selecting the parameter that maximizes the test-set metric makes the reported 0.52/0.78 optimistic estimates of external validity; this is a selection-on-the-test-set problem. Please use a proper model-selection procedure (e.g., a separate validation split or cross-validation) and report the performance of the chosen configuration on a held-out test set. Additionally, clarify whether the reported ARI/NMI include the 722 outlier entries; the note in the confusion-matrix section says outliers were excluded from that analysis, and if they are also excluded from ARI/NMI, the agreement is measured on only 613 of 1,335 (45.9%) entries. Report ARI/NMI both with outliers (e.g., treating -1 as an additional cluster) and without, so the coverage of the external validation is transparent.
- [Preprocessing; Data Source: NIH NLM CDE Repository] The preprocessing section states that 'each designation for a given CDE was represented as a distinct row in the final table,' while the data source section reports 'totaling 24,363 CDEs.' If a CDE can have multiple designations, the dataset contains rows at the designation level, not unique CDE level, and the clustering/classification/outlier statistics are computed on these rows. The paper then refers to '17,973 out of 24,363 CDEs' as outliers, which conflates rows with CDEs. This ambiguity affects the central quantitative claims: the number of clusters, the outlier percentage, and the classification accuracy could change if the analysis were performed on unique CDEs or if multiple designations were aggregated. Please clarify whether 24,363 refers to unique CDEs or total rows, and if multiple designations occur, explain how the analysis handles them or justify treating each designation as an independent item.
minor comments (5)
- [Evaluation; Table 7] The NMI range sentence is corrupted: 'The NMI scores ranged from 0.0604 to 0.8009 0.71 to 0.81' should be a single coherent range; please fix.
- [Evaluation; Table 7] At min_cluster_size=10 the ARI is 0.6176 and NMI is 0.8009, both higher than at the selected value of 7; the text says size 7 was chosen as balanced but does not explain why the better external metrics at size 10 were not preferred. Please justify the selection.
- [References] Reference 13 is a URL to a bachelor thesis PDF, and references 32 and 33 appear to be the same document; please replace with proper peer-reviewed citations or remove duplicates.
- [Figure 1 caption] The caption ends with '..' which appears to be a typo.
- [Cluster Labeling] The claim that clusters are 'meaningful' is currently supported only by internal clustering metrics and unvalidated LLM-generated labels; consider adding a human evaluation of label quality or a comparison to existing CDE classifications for at least a subset of clusters.
Circularity Check
The 90.46% classifier accuracy is a self-consistency check: the target labels come from HDBSCAN applied to the same embeddings used as features, so the result does not test assignment of new or unclustered CDEs.
-
fitted input called prediction
[Methods, Classification; Evaluation, Classification Evaluation Using LLM-Derived Labels from NIH CDEs]
"For each data point, a 1536-dimensional embedding vector was extracted, and used to form the feature matrix. The corresponding cluster labels were utilized as the target variable. ... In contrast to the clustering evaluation, this classification task does not rely on external ground truth. Instead, the cluster labels—automatically generated by a LLM via summarization of NIH CDEs—serve as the target variable. ... The classification model achieved an overall accuracy of 0.9046, demonstrating strong performance across several categories."
The classifier's target labels are produced by HDBSCAN from the very same 1,536-dimensional embedding vectors that form the Random Forest feature matrix. High accuracy therefore largely measures whether HDBSCAN clusters are separable in the embedding space that was used to define them, not whether the clusters are semantically valid or whether new or unclustered CDEs can be correctly routed. Because 17,973 of the 24,363 NIH CDEs (73.8%) were labeled as noise by HDBSCAN and are absent from the label set, the 90.46% figure covers only the 6,390 clustered CDEs, so it does not evaluate the paper's stated goal of assigning new or unclustered CDEs to labeled clusters.
full rationale
The framework is not globally circular: the Gravity Projects SDOH evaluation provides an independent external anchor, since ARI/NMI compare HDBSCAN clusters against curated SDOH domain labels rather than against labels generated by the same embeddings. That independent evidence keeps the paper from collapsing. However, the headline classification accuracy is a self-consistency result: the cluster labels used as targets are themselves outputs of unsupervised clustering on the same embedding vectors used as classifier inputs, and all HDBSCAN outliers (73.8% of the NIH repository) are excluded from the evaluation. The same coverage issue affects the Gravity evaluation, where 54.1% of points are outliers and the confusion matrix explicitly excludes them, and the min_cluster_size=7 choice appears in a sweep that reports ARI/NMI, so the reported agreement values may be optimistically selected if that sweep influenced the final choice. These are partial circularity and selection concerns, not a complete reduction of the method to its inputs.
Assumptions & free parameters
free parameters (3)
- min_cluster_size for NIH NLM CDEs =
20
- min_cluster_size for Gravity SDOH data =
7
- Number of CDEs per cluster used for LLM labeling =
20
assumptions (5)
- domain assumption OpenAI text-embedding-3-small embeddings capture enough semantic similarity to support CDE harmonization grouping.
- domain assumption HDBSCAN clusters in embedding space correspond to harmonization-relevant semantic groups.
- domain assumption Gravity Project's 21 SDOH domain labels are a valid external ground truth for semantic grouping.
- domain assumption LLM-generated cluster labels are accurate enough to serve as classification targets.
- domain assumption CDEs labeled as noise by HDBSCAN can be treated as outliers rather than as evidence that clustering failed.
Cite this review
Pith. "Pith review of A Dynamic Framework for Semantic Grouping of Common Data Elements (CDE) Using Embeddings and Clustering." pith.science (2026). https://pith.science/paper/PUY4YAMA
@misc{pith2026250602160,
author = {Pith},
title = {Pith review of: A Dynamic Framework for Semantic Grouping of Common Data Elements (CDE) Using Embeddings and Clustering},
year = {2026},
howpublished = {\url{https://pith.science/paper/PUY4YAMA}},
note = {Machine review of arXiv:2506.02160}
}
read the original abstract
This research aims to develop a dynamic and scalable framework to facilitate harmonization of Common Data Elements (CDEs) across heterogeneous biomedical datasets by addressing challenges such as semantic heterogeneity, structural variability, and context dependence to streamline integration, enhance interoperability, and accelerate scientific discovery. Our methodology leverages Large Language Models (LLMs) for context-aware text embeddings that convert CDEs into dense vectors capturing semantic relationships and patterns. These embeddings are clustered using Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) to group semantically similar CDEs. The framework incorporates four key steps: (1) LLM-based text embedding to mathematically represent semantic context, (2) unsupervised clustering of embeddings via HDBSCAN, (3) automated labeling using LLM summarization, and (4) supervised learning to train a classifier assigning new or unclustered CDEs to labeled clusters. Evaluated on the NIH NLM CDE Repository with over 24,000 CDEs, the system identified 118 meaningful clusters at an optimized minimum cluster size of 20. The classifier achieved 90.46 percent overall accuracy, performing best in larger categories. External validation against Gravity Projects Social Determinants of Health domains showed strong agreement (Adjusted Rand Index 0.52, Normalized Mutual Information 0.78), indicating that embeddings effectively capture cluster characteristics. This adaptable and scalable approach offers a practical solution to CDE harmonization, improving selection efficiency and supporting ongoing data interoperability.
Reference graph
Works this paper leans on
-
[1]
Kush, R. D. et al. FAIR data sharing: The roles of common data elements and harmonization. J. Biomed. Inform. 107 , 103421 (2020)
work page 2020
-
[2]
ISO/IEC 11179, Information Technology - Metadata Registries (MDR)
ISO/IEC JTC1 SC32. ISO/IEC 11179, Information Technology - Metadata Registries (MDR) . (2007)
work page 2007
-
[3]
Metadata Registry https://dss.aristotlecloud.io/help/concepts/aristotle_mdr/valuedomain
Concept help - Value Domain. Metadata Registry https://dss.aristotlecloud.io/help/concepts/aristotle_mdr/valuedomain
-
[5]
https://datascience.cancer.gov/resources/metadata
Metadata Services for Cancer Research. https://datascience.cancer.gov/resources/metadata
-
[6]
https://metadatacenter.readthedocs.io/en/latest/
CEDAR. https://metadatacenter.readthedocs.io/en/latest/
-
[7]
Hamilton, C. M. et al. The PhenX Toolkit: get the most from your measures. Am. J. Epidemiol. 174 , 253–260 (2011)
work page 2011
-
[8]
https://meteor.aihw.gov.au/content/181414
About METEOR. https://meteor.aihw.gov.au/content/181414
-
[9]
Hu, Z. et al. Toward better semantic interoperability of data element repositories in medicine: Analysis study. JMIR Med. Inform. 12 , e60293 (2024)
work page 2024
Show all 35 references
- [10]
-
[11]
& Astels, S
McInnes, L., Healy, J. & Astels, S. hdbscan: Hierarchical density based clustering. J. Open Source Softw. 2 , 205 (2017)
2017
-
[12]
& Tian, Y
Xu, D. & Tian, Y. A comprehensive survey of clustering algorithms. Ann. Data Sci. 2 , 165–193 (2015). 13. https://rucforsk.ruc.dk/ws/portalfiles/portal/64939887/Machine_Learning_Bachelor_2019.pdf
2015
-
[14]
Dimensionality Reduction : PCA, tSNE, UMAP
Vijay, H. Dimensionality Reduction : PCA, tSNE, UMAP. Auriga IT https://aurigait.com/blog/blog-easy-explanation-of-dimensionality-reduction-and-techniques/ (2023)
2023
-
[15]
Raiaan, M. A. K. et al. A review on large language models: Architectures, applications, taxonomies, open issues and challenges. IEEE Access 12 , 26839–26874 (2024)
2024
-
[16]
& Stryker, C
Bergmann, D. & Stryker, C. What is Vector Embedding? https://www.ibm.com/think/topics/vector-embedding (2025)
2025
-
[17]
Wang, Y. et al. CDEMapper: Enhancing NIH common data element normalization using large language models. arXiv [cs.IR] (2024)
2024
-
[18]
Renner, R. et al. Using an artificial neural network to map cancer common data elements to the biomedical research integrated domain group model in a semi-automated manner. BMC Med. Inform. Decis. Mak. 19 , 276 (2019)
2019
-
[19]
Hao, X. et al. Mapping of Alzheimer’s disease related data elements and the NIH Common Data Elements. BMC Med. Inform. Decis. Mak. 24 , 103 (2024)
2024
-
[20]
National Institute on Aging https://www.nia.nih.gov/research/dab-dbsr-dgcg-dn/workshops/common-data-elements-web inar-series
Common Data Elements Webinar Series. National Institute on Aging https://www.nia.nih.gov/research/dab-dbsr-dgcg-dn/workshops/common-data-elements-web inar-series
-
[21]
https://www.icpsr.umich.edu/web/sbeccc/cms/4804
Common Data Elements. https://www.icpsr.umich.edu/web/sbeccc/cms/4804
-
[22]
Practical Data Stewardship
Plotkin, D. Practical Data Stewardship. in Data Stewardship 97–126 (Elsevier, 2014). doi:10.1016/b978-0-12-410389-4.00006-4
2014 doi
-
[23]
https://cde.nlm.nih.gov/home
NIH Common Data Elements (CDE) Repository. https://cde.nlm.nih.gov/home
-
[24]
https://www.commondataelements.ninds.nih.gov/
Welcome. https://www.commondataelements.ninds.nih.gov/
-
[25]
Weissman, A. et al. Development and implementation of the National Heart, Lung, and Blood Institute COVID-19 common data elements. J. Clin. Transl. Sci. 6 , e142 (2022)
2022
-
[26]
Rubinstein, Y. R. & McInnes, P. NIH/NCATS/GRDR® Common Data Elements: A leading force for standardized data collection. Contemp. Clin. Trials 42 , 78–80 (2015)
2015
-
[27]
https://www.healthmeasures.net/explore-measurement-systems/promis
PROMIS. https://www.healthmeasures.net/explore-measurement-systems/promis
-
[28]
https://www.healthmeasures.net/explore-measurement-systems/neuro-qol
Neuro-QoL. https://www.healthmeasures.net/explore-measurement-systems/neuro-qol
-
[29]
https://platform.openai.com/docs/guides/embeddings/
OpenAI Platform. https://platform.openai.com/docs/guides/embeddings/
-
[30]
Fiveable https://library.fiveable.me/modern-statistical-prediction-and-machine-learning/unit-14/evalua tion-metrics-unsupervised-learning-clustering/study-guide/UuzjLvrGcbnj7uhw
Evaluation Metrics for Unsupervised Learning and Clustering. Fiveable https://library.fiveable.me/modern-statistical-prediction-and-machine-learning/unit-14/evalua tion-metrics-unsupervised-learning-clustering/study-guide/UuzjLvrGcbnj7uhw
-
[31]
& Jacob, J
Pareek, J. & Jacob, J. Data Compression and Visualization Using PCA and T-SNE. in Advances in Information Communication Technology and Computing 327–337 (Springer Singapore, Singapore, 2021). doi:10.1007/978-981-15-5421-6_34
2021 doi
-
[33]
A Consolidated Overview of Evaluation and Performance Metrics for Machine Learning and Computer Vision
-
[34]
https://catalyst.nejm.org/doi/full/10.1056/cat.17.0312
-
[35]
The Gravity Project
Lousberg, C. The Gravity Project. https://confluence.hl7.org/spaces/GRAV/overview
-
[36]
Terminology Workstream - Gravity Project - Confluence
Lousberg, C. Terminology Workstream - Gravity Project - Confluence. https://confluence.hl7.org/spaces/GRAV/pages/91994234/Terminology+Workstream
-
[37]
Confusion Matrix Visualization
Susmaga, R. Confusion Matrix Visualization. in Intelligent Information Processing and Web Mining 107–116 (Springer Berlin Heidelberg, Berlin, Heidelberg, 2004). doi:10.1007/978-3-540-39985-8_12
2004 doi
-
[38]
scikit-learn https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
classification_report. scikit-learn https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.