{"id":"952c5f32-1f39-4462-ad14-6751e80f2c73","arxiv_id":"2506.02160","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM embeddings clustered with HDBSCAN group 6,390 NIH common data elements into 118 semantic clusters, and a random forest on the same embeddings reaches 90.46% accuracy on cluster labels.","lead":"This paper builds a pipeline that converts common data elements from biomedical repositories into LLM embeddings, clusters them with HDBSCAN, and trains a classifier to assign new elements to the resulting groups. A reader interested in data harmonization would learn whether automated semantic grouping can replace manual curation, with the caveat that three quarters of the data were treated as outliers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 90.46% accuracy is measured only on the 26.2% of CDEs that HDBSCAN clustered; no evaluation covers the 73.8% outliers, so the stated ability to assign new or unclustered CDEs is unsupported.","rationale":"The paper's abstract and conclusion present 90.46% accuracy and 118 clusters as evidence that the framework supports harmonization of the NIH NLM repository. The reader's conditional verdict identifies the outlier fraction as the weakest premise; I agree and would sharpen it: the outlier exclusion is not merely a coverage caveat but affects the validity of the accuracy claim as a measure of the deployment scenario, because the classifier never sees the majority population it is supposed to handle. The external Gravity validation is the strongest independent evidence, but its min_cluster_size selection after viewing ARI/NMI and the ambiguity about outlier handling in the metrics make it weaker than the abstract implies. These are correctable with additional experiments, so the verdict remains CONDITIONAL rather than REJECT; the paper should either narrow its claims to clustered CDEs or supply the missing full-corpus and outlier-aware evaluations.","tokens_in":17239,"tokens_out":7091,"duration_ms":66877,"concrete_test":"Compute an operational accuracy over the full NIH NLM repository: add HDBSCAN's -1 label as a 119th class (or add a distance-based reject option to the classifier) and report accuracy and macro-F1 on all 24,363 CDEs under the same 80/20 split, plus a separate report for the 17,973 outlier CDEs. For the Gravity dataset, recompute ARI/NMI with all 722 outliers included in the comparison rather than excluded. If full-corpus accuracy is close to the 73.8% outlier majority-class baseline, or if including outliers drops ARI/NMI noticeably below 0.52/0.78, then the reported metrics describe only the clustered subset and do not support the claimed ability to assign new or unclustered CDEs.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing concern is that the 90.46% accuracy figure does not measure the framework's stated purpose: assigning new or unclustered CDEs to labeled groups. The classifier is trained and tested only on the 6,390 non-outlier NIH CDEs assigned to the 118 clusters; the 17,973 CDEs (73.8%) labeled as noise by the same HDBSCAN run are absent from the label set. Because the target labels are created by HDBSCAN from embeddings that also serve as classifier features, the high test accuracy mostly shows separability of the embedding space with respect to the clusters it produced, not that clusters are meaningful harmonization targets or that a new/unclustered CDE can be routed correctly. The intended use case is precisely the unclustered population: on the NIH repository 73.8% of items are outliers, and on the Gravity validation set 54.1% are outliers (Table 7). No experiment reports accuracy on held-out outliers, accuracy on the full repository with an outlier class, or precision of forced cluster assignment for outliers. The Gravity external validation has the same coverage ambiguity if outliers are excluded from ARI/NMI, and min_cluster_size=7 was selected after inspecting those same external metrics, so 0.52/0.78 are optimistic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end framework for harmonizing Common Data Elements (CDEs): (1) generate LLM embeddings (OpenAI text-embedding-3-small) of CDE text, (2) cluster the embeddings with HDBSCAN, (3) assign cluster labels via GPT-3.5-turbo summarization, and (4) train a Random Forest classifier to assign new or unclustered CDEs to these labels. The framework is evaluated on the NIH NLM CDE Repository (24,363 entries) and externally on the Gravity Projects SDOH dataset. The authors report 118 clusters at min_cluster_size=20, a classification accuracy of 90.46%, and external validation with ARI 0.52 and NMI 0.78. The conclusion acknowledges that 73.8% of NIH entries and 54.1% of Gravity entries were labeled outliers by HDBSCAN.","tokens_in":17501,"tokens_out":4260,"duration_ms":39465,"significance":"If the reported results were valid as stated, the framework would offer a practical, scalable tool for suggesting harmonization candidates across CDE repositories, with a clear pipeline and reproducible supplementary code. The use of HDBSCAN to handle noise and the inclusion of external validation data are commendable. However, several evaluation gaps directly affect the central claims: the accuracy figure applies only to the 26.2% of entries that formed clusters, the classification evaluation is largely circular because the same embeddings generate both the cluster labels and the classifier features, and the Gravity validation parameter was selected after inspecting the external metrics. These issues currently prevent the conclusions from being supported at the level claimed.","major_comments":[{"comment":"The 90.46% accuracy is computed only on entries assigned to the 118 HDBSCAN clusters, i.e., 6,390 of 24,363 (26.2%), while 17,973 entries (73.8%) are outliers. The abstract and conclusion state an 'overall accuracy' without this restriction, and the classifier is described as a mechanism to assign 'new or unclustered CDEs' to labeled clusters. No experiment evaluates classification of the outlier population, either as a separate class or via forced assignment. Because the outlier population is the majority of the repository, the reported accuracy does not support the framework's stated utility for the full data distribution. Please report metrics on the full dataset (including an outlier/reject option) or clearly qualify the accuracy as applying only to the non-outlier subset, and discuss how the intended use case handles the 73.8% outliers.","section":"Abstract; Classification Evaluation Using LLM-Derived Labels from NIH CDEs; Conclusion"},{"comment":"The classification evaluation is substantially circular. HDBSCAN assigns cluster labels from the embeddings; the Random Forest is then trained to predict those same cluster labels from the same embeddings. High test accuracy therefore mainly demonstrates that the clusters are separable in the embedding space with respect to the labels HDBSCAN produced, not that the clusters correspond to semantically meaningful harmonization groups or that a new/unclustered CDE would be routed to a valid group. An independent ground truth is needed. As a concrete remedy, evaluate classification on the Gravity SDOH dataset using the known SDOH domain labels as targets, and/or provide a human-annotated gold standard for a sample of NIH CDEs. Without such an evaluation, the '90.46% accuracy' claim is not evidence of semantic grouping quality.","section":"Classification; Classification Evaluation Using LLM-Derived Labels from NIH CDEs"},{"comment":"The choice of min_cluster_size=7 for the Gravity dataset appears to have been made after inspecting the external ARI/NMI values across candidate sizes: min_cluster_size=7 gives ARI 0.5239 and NMI 0.7768, while min_cluster_size=10 gives ARI 0.6176 and NMI 0.8009. Selecting the parameter that maximizes the test-set metric makes the reported 0.52/0.78 optimistic estimates of external validity; this is a selection-on-the-test-set problem. Please use a proper model-selection procedure (e.g., a separate validation split or cross-validation) and report the performance of the chosen configuration on a held-out test set. Additionally, clarify whether the reported ARI/NMI include the 722 outlier entries; the note in the confusion-matrix section says outliers were excluded from that analysis, and if they are also excluded from ARI/NMI, the agreement is measured on only 613 of 1,335 (45.9%) entries. Report ARI/NMI both with outliers (e.g., treating -1 as an additional cluster) and without, so the coverage of the external validation is transparent.","section":"Evaluation; Table 7"},{"comment":"The preprocessing section states that 'each designation for a given CDE was represented as a distinct row in the final table,' while the data source section reports 'totaling 24,363 CDEs.' If a CDE can have multiple designations, the dataset contains rows at the designation level, not unique CDE level, and the clustering/classification/outlier statistics are computed on these rows. The paper then refers to '17,973 out of 24,363 CDEs' as outliers, which conflates rows with CDEs. This ambiguity affects the central quantitative claims: the number of clusters, the outlier percentage, and the classification accuracy could change if the analysis were performed on unique CDEs or if multiple designations were aggregated. Please clarify whether 24,363 refers to unique CDEs or total rows, and if multiple designations occur, explain how the analysis handles them or justify treating each designation as an independent item.","section":"Preprocessing; Data Source: NIH NLM CDE Repository"}],"minor_comments":[{"comment":"The NMI range sentence is corrupted: 'The NMI scores ranged from 0.0604 to 0.8009 0.71 to 0.81' should be a single coherent range; please fix.","section":"Evaluation; Table 7"},{"comment":"At min_cluster_size=10 the ARI is 0.6176 and NMI is 0.8009, both higher than at the selected value of 7; the text says size 7 was chosen as balanced but does not explain why the better external metrics at size 10 were not preferred. Please justify the selection.","section":"Evaluation; Table 7"},{"comment":"Reference 13 is a URL to a bachelor thesis PDF, and references 32 and 33 appear to be the same document; please replace with proper peer-reviewed citations or remove duplicates.","section":"References"},{"comment":"The caption ends with '..' which appears to be a typo.","section":"Figure 1 caption"},{"comment":"The claim that clusters are 'meaningful' is currently supported only by internal clustering metrics and unvalidated LLM-generated labels; consider adding a human evaluation of label quality or a comparison to existing CDE classifications for at least a subset of clusters.","section":"Cluster Labeling"}],"recommendation":"major_revision","confidential_remarks":"The paper has a clear and reproducible pipeline, and the general direction is of interest to the CDE harmonization community. The load-bearing issues are the coverage of the accuracy claim, the circular classification evaluation, and the selection-on-test-set for the Gravity validation. These are fixable with additional experiments and reanalysis, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—here is my read on 2506.02160. The paper is a workmanlike end-to-end pipeline: LLM embeddings (text-embedding-3-small), HDBSCAN clustering, LLM-generated cluster labels, and a Random Forest classifier, applied to 24,363 CDEs from the NIH NLM repository, with an external validation against Gravity Projects' SDOH domains. What is genuinely new is the specific assembly of these ingredients for CDE grouping, and the external SDOH comparison is a real attempt to check whether the clusters correspond to something outside the embeddings. The authors also disclose several limitations in the conclusion, including the 73.8% outlier rate, and they promise supplementary code and data.\n\nThe soft spots are real and center on the main claim. The 90.46% accuracy is computed only on the 6,390 CDEs that HDBSCAN assigned to clusters, not on the 17,973 outliers. The abstract says the classifier 'assigns new or unclustered CDEs to labeled clusters,' but that is precisely the population that is never tested. No experiment reports accuracy on held-out outliers, or on the full repository with an outlier class. So the headline number does not support the stated use case.\n\nSecond, the classification evaluation is largely circular. Cluster labels come from HDBSCAN applied to the embeddings, and the classifier predicts those labels from the same embeddings. High accuracy mostly shows that clusters are separable in the embedding space, not that they are semantically meaningful harmonization targets. That is a candidate-suggestion tool, not a validation of the groupings.\n\nThird, the Gravity validation has a parameter-selection problem. The min_cluster_size of 7 was chosen after inspecting ARI/NMI across the table, so the reported 0.52/0.78 are optimistic, and outliers were excluded from the confusion matrix. Minor point: no error bars or baseline comparisons (e.g., random embeddings or term-matching) are reported.\n\nNone of this kills the paper. The pipeline is sensible and the authors are honest in the conclusion about the outlier issue. But the abstract oversells, and the evidence as presented is not sufficient for the central claim. This is a CONDITIONAL to REVISE story, not a reject.\n\nRecommendation: send it to peer review. A serious referee should ask for accuracy on the full repository, baselines, and a defense of the parameter choice. With those revisions it could be a useful methods paper for CDE harmonization practitioners. I would not cite it in my own work in its current form, but someone working on metadata interoperability should be aware of it.","headline":"Useful end-to-end pipeline for CDE grouping, but the headline accuracy is measured only on the 26% of CDEs that clustered, so the 'assign new/unclustered CDEs' claim is unsupported.","tokens_in":18065,"tokens_out":3162,"would_cite":false,"duration_ms":24795,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Embedding CDE text and clustering with HDBSCAN yields 118 semantic clusters and 90.46% classification accuracy.","keywords":["Common Data Elements (CDE)","CDE harmonization","LLM embeddings","HDBSCAN clustering","semantic grouping","biomedical metadata","Random Forest classification","Gravity Project SDOH"],"falsifier":"Take the 17,973 outlier CDEs, embed them with the same model, and count how many lie closer to one of the 118 cluster centroids than the median within-cluster distance of that cluster. If a large fraction (for example, more than a quarter) are that close, the 'outliers are semantically unique' premise fails and the reported coverage overstates what the framework can group.","tokens_in":17027,"feed_emoji":"🧬","tokens_out":9803,"duration_ms":94786,"temperature":0.7,"pith_summary":"Common Data Elements are standardized questions and answer sets that biomedical studies reuse, but the same concept is often phrased differently across repositories, making harmonization a mostly manual chore. This paper proposes a four-step pipeline—LLM text embeddings, HDBSCAN clustering, LLM-generated cluster labels, and a trained classifier—to automate the grouping of semantically similar CDEs. On the NIH NLM CDE Repository (24,363 CDEs) the pipeline reports 118 meaningful clusters at a minimum cluster size of 20, and the classifier reproduces the cluster labels with 90.46% accuracy. External validation against the Gravity Projects' Social Determinants of Health domains gives an Adjusted Rand Index of 0.52 and Normalized Mutual Information of 0.78, which the paper reads as strong agreement. The paper's own caveat is important: 17,973 of the 24,363 CDEs (73.8%) were labeled as outliers, so the demonstrated grouping covers only the densest semantic regions.","feed_headline":"Clustering pipeline groups biomedical data elements with 90% accuracy","feed_subtitle":"LLM embeddings plus HDBSCAN sort similar CDEs into labeled clusters, cutting manual harmonization work.","key_machinery":"The load-bearing object is the dense vector embedding: each CDE's designation, definition, and permissible values are concatenated into a single string and passed through OpenAI's text-embedding-3-small model to produce a 1,536-dimensional vector in which semantically similar CDEs land close together. HDBSCAN (hierarchical density-based spatial clustering of applications with noise) then groups these vectors into clusters of arbitrary shape and density without requiring the number of clusters in advance, while labeling points that do not fit as outliers ($-1$). Cluster labels come from GPT-3.5-turbo summarization of up to 20 representative CDEs per cluster. A Random Forest classifier using the same embeddings as features learns to predict those cluster labels, turning the embedding space into an operational routing tool for new or previously unclustered CDEs.","core_discovery":"The central claim is that semantic heterogeneity among Common Data Elements can be addressed by representation rather than by hand-built mapping rules. Concatenating each CDE's preferred designation, definition, and permissible values into one text string, embedding that string with OpenAI's text-embedding-3-small model, and clustering the resulting 1,536-dimensional vectors with HDBSCAN yields groups that correspond to recognizable biomedical and social concepts. With min_cluster_size set to 20, the framework found 118 clusters in the NIH NLM repository; a Random Forest trained on the embeddings assigned held-out CDEs to the correct LLM-generated cluster label with 90.46% overall accuracy; and the same method applied to Gravity Projects SDOH data produced clusters that matched the 21 known domains at ARI 0.52 and NMI 0.78. The framework is offered as a scaffold for harmonization—suggesting which CDEs belong together—rather than as a replacement for formal semantic mapping.","pith_inferences":["Beyond the paper: a stricter test would compare the LLM-generated cluster labels against a human-curated medical ontology to see whether the 118 clusters correspond to recognized concepts, not just to coherent text clusters.","Beyond the paper: the 73.8% outlier rate suggests a two-pass design—cluster first, then re-embed the outliers with a smaller minimum cluster size or assign them to the nearest existing cluster by a cosine-similarity threshold—to recover rare but meaningful subconcepts the single pass discards.","Beyond the paper: if the same embedding space were applied across multiple CDE repositories at once, it could expose cross-repository synonyms that the paper only demonstrates within one repository plus a single external benchmark."],"forward_implications":["New CDEs entering a repository can be assigned to an existing labeled cluster automatically, so curators see a suggested grouping immediately rather than starting from scratch.","CDEs from different steward organizations (LOINC, NCI, NHLBI, GRDR, and others) land in the same cluster when they mean the same thing, which the 'Educational Attainment Levels' example illustrates.","The external validation against SDOH domains indicates the same embedding-plus-HDBSCAN recipe transfers to at least one other CDE collection with known domain labels.","For well-populated semantic categories the classifier is reliable, but the paper's own class-level report shows that small or sparse clusters can collapse to zero precision and recall.","The 118 clusters provide a concrete shortlist of candidate CDEs for harmonization, filling a gap the paper identifies in earlier repository-alignment work."],"supporting_citations":[{"why":"Supplies the dataset of 24,363 CDEs from the NIH NLM CDE Repository used for clustering and classification.","marker":"23"},{"why":"Supplies the OpenAI text-embedding-3-small model that generates the 1,536-dimensional semantic vectors.","marker":"29"},{"why":"Supplies HDBSCAN, the density-based clustering algorithm that forms the clusters and labels outliers.","marker":"11"},{"why":"Supplies the Gravity Project as the external benchmark whose SDOH domains serve as proxy ground truth.","marker":"35"},{"why":"Supplies the 21 SDOH domain definitions used for the ARI and NMI external validation.","marker":"36"}],"fun_headline_variants":["LLM embeddings cluster 24k clinical data elements into 118 groups","Semantic grouping of clinical data elements hits 90% accuracy","Embedding-based clustering groups 24k CDEs with 90% accuracy","HDBSCAN + LLM embeddings sort 24k biomedical CDEs with 90% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the roughly three-quarters of repository CDEs that HDBSCAN labeled as outliers are genuinely semantically unique or unimportant, so that the 118 clusters and the 90.46% accuracy—both computed on the clustered subset—describe the framework's real coverage of the full repository.","fun_headline_variants_meta":{"raw":{"variants":["LLM embeddings cluster 24k clinical data elements into 118 groups","Semantic grouping of clinical data elements hits 90% accuracy","Embedding-based clustering groups 24k CDEs with 90% accuracy","HDBSCAN + LLM embeddings sort 24k biomedical CDEs with 90% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000782,"raw_usage":{"total_tokens":3488,"prompt_tokens":1016,"completion_tokens":2472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":2388}},"tokens_in":632,"tokens_out":2472,"duration_ms":18824,"temperature":1.0,"reasoning_tokens":2388,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:28:46.467802+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 17,973 outlier CDEs, embed them with the same model, and count how many lie closer to one of the 118 cluster centroids than the median within-cluster distance of that cluster. If a large fraction (for example, more than a quarter) are that close, the 'outliers are semantically unique' premise fails and the reported coverage overstates what the framework can group.","supporting_citations":[{"cited_title":"https://cde.nlm.nih.gov/home","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset of 24,363 CDEs from the NIH NLM CDE Repository used for clustering and classification."},{"cited_title":"https://platform.openai.com/docs/guides/embeddings/","cited_arxiv_id":null,"evidence_quote":"Supplies the OpenAI text-embedding-3-small model that generates the 1,536-dimensional semantic vectors."},{"cited_title":"& Astels, S","cited_arxiv_id":null,"evidence_quote":"Supplies HDBSCAN, the density-based clustering algorithm that forms the clusters and labels outliers."},{"cited_title":"The Gravity Project","cited_arxiv_id":null,"evidence_quote":"Supplies the Gravity Project as the external benchmark whose SDOH domains serve as proxy ground truth."},{"cited_title":"Terminology Workstream - Gravity Project - Confluence","cited_arxiv_id":null,"evidence_quote":"Supplies the 21 SDOH domain definitions used for the ARI and NMI external validation."}],"review_version":1}