REVIEW 4 major objections 5 minor 23 references
ICU-TSB: A Benchmark for Temporal Patient Representation Learning for Unsupervised Stratification into Patient Cohorts
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Temporal representation learning can rediscover clinically meaningful ICU patient cohorts from unlabeled time series, reaching v-measure up to 0.46 at the coarsest ICD level and up to 0.40 at the finest, according to a new benchmark.
desk verdict A useful benchmark artifact whose reported scores need one clarification—the Optuna objective—before they can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the ICD-9-CM hierarchy as a rooted tree with four levels, in which each patient code at level $i$ has exactly one parent at level $i-1$. The evaluation pipeline encodes each ICU stay as an embedding (the last hidden state of an autoregressive LSTM or GRU, or concatenated statistical moments), clusters the embeddings with k-Means, and compares the resulting clusters with the patients' ICD codes at each level using v-measure, adjusted mutual information, and accuracy. A second mechanism is hierarchical rediscovery: patients are first clustered at the broadest level, and each cluster is re-clustered at the next finer level, mirroring the taxonomy tree. Interpretability is handled by labeling each cluster with the centroid patient's code, the medoid patient's code, or the majority code among its members.
What would settle it
Run the same pipeline with ICD labels randomly permuted: v-measure and adjusted mutual information should fall to chance levels, and they should stay low when clusters are transferred across datasets with different coding practices. More directly, compare the discovered clusters against expert-adjudicated phenotypes or against survival, treatment, and lab-value differences; if high-scoring clusters show no clinical separation, the benchmark is measuring alignment with coding conventions rather than clinical meaning.
Extended reading notes
Core claim
The central claim is that self-supervised temporal representation learning can rediscover clinically meaningful ICU patient cohorts without using outcome labels, and that this capability can be measured as a hierarchy-rediscovery task over the ICD-9-CM taxonomy. The paper reports that LSTM and GRU embeddings, clustered with k-Means, align with ICD codes at all four hierarchy levels, with v-measure, a cluster-quality metric combining homogeneity and completeness, reaching up to 0.46 at the top level $L_1$ and up to 0.40 at the bottom level $L_4$, generally beating a statistical baseline built from time-window moments. The authors also find that majority-vote labeling gives the most reliable interpretable cluster names, and that hierarchical re-clustering from broad to specific levels captures coarse disease categories more reliably than fine-grained ones.
Load-bearing premise
The evaluation treats the top-25 most frequent ICD-9-CM codes in each dataset as the true clinically meaningful cohort labels; if those codes are noisy, billing-driven, or too heterogeneous, the v-measure scores measure coding consistency rather than clinical usefulness.
Editorial extensions
If this is right
- Self-supervised temporal embeddings can discover coarse disease cohorts from ICU time series without relying on labeled outcomes.
- Recurrent architectures (LSTM and GRU) beat a statistical-moments baseline in most datasets and hierarchy levels, so nonlinear temporal modeling carries real signal for stratification.
- The hierarchical ICD scoring procedure gives future models a common, reproducible yardstick for unsupervised stratification.
- Fine-grained stratification at lower ICD levels remains unreliable, so progress is most needed there.
- Majority-vote labeling is the most reliable way to turn a discovered cluster into an interpretable diagnostic label.
Reading between the lines
- If ICD codes are billing-driven or too heterogeneous, the reported v-measure may reward coding consistency rather than clinical subtype validity; an external check against adjudicated phenotypes or outcome differences would sharpen the claim.
- The same hierarchy-rediscovery recipe could be applied to other ontologies, for example CCS or phecode groupings, and to multimodal EHR data, where fine-grained levels might become easier to recover.
- Because neural gains concentrate at coarse levels, contrastive or semi-supervised objectives that make embeddings invariant to nuisance temporal variation could plausibly push fine-level v-measure above the reported 0.40.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ICU-TSB, a benchmark for evaluating unsupervised patient stratification on three ICU datasets (eICU, MIMIC-IV, SiC). Patient trajectories are encoded using a statistical baseline, LSTM, and GRU; the resulting embeddings are clustered with k-Means and evaluated by how well clusters align with four levels of an ICD-9-CM hierarchy. The authors also propose a hierarchical clustering formulation and compare centroid-, medoid-, and majority-vote strategies for assigning interpretable labels. The main empirical claim is that temporal representation learning can rediscover clinically meaningful patient cohorts, with v-measure ranging up to 0.46 at the top level and up to 0.40 at the lowest level, though the task remains challenging.
Significance. If the claims hold, ICU-TSB would be a useful, reproducible resource for a relatively under-benchmarked problem. The paper's strengths are the use of three public datasets, the explicit hierarchical evaluation framework over disease taxonomies, and the public code repository, which make the experimental protocol checkable. The empirical comparison of several representation-learning and clustering options is relevant to practitioners. However, the benchmark's credibility depends on the integrity of the unsupervised protocol, and the current text leaves a key confound unresolved.
major comments (4)
- [Methods, Cluster label assignment (end)] The Optuna optimization described in the final paragraph of this section is the manuscript's most important unresolved point. The text states that hyperparameters, including the number of clusters and t-SNE parameters, are optimized with 50 trials per formulation, but it does not state the objective function. If the objective is any of the external metrics listed immediately after (v-measure, AMI, accuracy) computed against the same ICD labels used in Figures 4–6, then the reported rediscovery scores are fitted to the evaluation labels rather than measured in an unsupervised setting. This would directly undermine the abstract's central claim. The authors must specify the Optuna objective, state whether labels were accessible during optimization, and define a validation protocol that keeps labels quarantined; if label-based tuning was used, the evaluation must be repeated with held-out labels or with model selection based on internal criteria.
- [Results, Figures 4–6] No error bars, confidence intervals, or repeated-run information are reported anywhere in the Results. Since k-Means initialization, t-SNE, and neural-network training are all stochastic, statements such as 'LSTM models significantly surpassed STAT' and 'majority vote outperformed both metrics across datasets' are not supported by the evidence shown. The authors should report means and standard deviations over multiple seeds and, where appropriate, per-dataset significance tests or effect sizes.
- [Methods, Data preprocessing / Cluster label assignment] The manuscript refers to 'training sets' for RobustScaler normalization and for cluster-label assignment, but no train/validation/test split is ever defined. Without such a split, it is unclear whether the patients used for hyperparameter selection, normalization, or labeling are the same as those used for the final evaluation. The benchmark should specify the exact split, including whether it is at the patient or ICU-stay level, and explain how the split interacts with the Optuna search and with the hierarchical clustering recursion.
- [Dataset description / ICD hierarchical tree levels] The entire evaluation treats administrative ICD-9-CM and ICD-10 codes as a proxy for 'clinically meaningful patient cohorts,' but the paper does not assess the validity of this proxy. Billing codes can be noisy, inconsistent across institutions, and influenced by reimbursement patterns. Even if the optimization-protocol confound is resolved, the reported scores would quantify alignment with coding systems, not necessarily with clinical subtypes. The authors should state this assumption explicitly and provide at least a qualitative or external validation (e.g., comparison with recognized phenotype definitions or manual chart review) to support the benchmark's clinical meaningfulness claim.
minor comments (5)
- [Abstract / Dataset description] There is a typographical error in the abstract ('v-measuring' should be 'v-measure'), and in the Dataset description section 'the the hierarchy' contains a duplicated article.
- [Methods, Temporal patient representation learning] The STAT baseline is described primarily by a citation to Proios et al. (2023); the specific time-window lengths and statistical moments should be stated in the text or explicitly identified as the configuration shipped in the repository.
- [Methods, ICD hierarchical tree levels] The manuscript mentions CCS codes in the Introduction, Figure 4, and Figure 6, but the CCS taxonomy is never defined or related to the ICD hierarchy; add a definition and the mapping procedure used.
- [Experimental setting] The LSTM and GRU architectures are under-specified: hidden size, number of layers, sequence length, missing-value handling, and number of training epochs are not given. Either include these values or state that the exact configuration is defined in the repository and point to the relevant file.
- [Figures 5 and 6] The captions for Figures 5 and 6 do not identify the clustering algorithm or, for Figure 6, which label-assignment variants are shown; the captions should be expanded so each panel can be mapped to the methods described in the text.
Circularity Check
No material circularity: ICD labels are used only for post-clustering evaluation, not for embedding training; the Optuna objective is unspecified but this is an unresolved leakage risk rather than a demonstrated circular reduction.
full rationale
The paper's central claim is an extrinsic benchmark result, not a derivation that presupposes its own conclusion. Patient embeddings are produced by autoregressive LSTMs/GRUs trained with MSE loss or by a statistical baseline; ICD labels are explicitly held out during clustering ('labels yp,i exist for each patient p, they remain unseen during the clustering process') and are used only afterward to compute v-measure, AMI, and accuracy. The reported v-measures (up to 0.46 at the top ICD level, up to 0.40 at the lowest) are modest, which is inconsistent with a benchmark whose scores are forced to the labels by construction. Self-citations to prior work by the same group (Proios et al. 2023 for the STAT baseline; Bornet et al. 2025 and Jaume-Santero et al. 2022 for context) describe baselines or background and are not load-bearing for the validity of the benchmark. One methodological ambiguity should be flagged: the Optuna hyperparameter search in the 'Cluster label assignment' section states 'We optimize hyperparameters (including number of clusters and t-SNE parameters) using optuna ... with 50 trials for each formulation of the problem' but does not state the objective function. If that objective were v-measure/AMI/accuracy against the same ICD labels later reported, the rediscovery scores would be inflated by transductive leakage. As written, the paper provides no evidence that this was avoided, but it also provides no equation or text showing that the reported scores are equivalent to the tuning objective. Under the rule that circularity must be demonstrated by quote and specific reduction rather than speculated, this is a reproducibility/leakage risk, not a confirmed circular step.
Assumptions & free parameters
free parameters (3)
- number_of_clusters_k =
optimized per dataset and level via Optuna
- t_sne_parameters =
not reported
- top_25_code_selection =
25 per dataset
assumptions (4)
- domain assumption ICD-9-CM codes are a valid extrinsic ground truth for clinically meaningful patient cohorts.
- domain assumption After preprocessing with ricu and hourly resampling, the three ICU datasets are sufficiently comparable for a multicentric benchmark.
- domain assumption Autoregressive next-step prediction yields representations that capture disease-relevant temporal patterns.
- domain assumption The mapping from ICD-10 to ICD-9 and the top-25 frequency filter preserve the hierarchy.
Cite this review
Pith. "Pith review of ICU-TSB: A Benchmark for Temporal Patient Representation Learning for Unsupervised Stratification into Patient Cohorts." pith.science (2026). https://pith.science/paper/DKI5TV7X
@misc{pith2026250606192,
author = {Pith},
title = {Pith review of: ICU-TSB: A Benchmark for Temporal Patient Representation Learning for Unsupervised Stratification into Patient Cohorts},
year = {2026},
howpublished = {\url{https://pith.science/paper/DKI5TV7X}},
note = {Machine review of arXiv:2506.06192}
}
read the original abstract
Patient stratification identifying clinically meaningful subgroups is essential for advancing personalized medicine through improved diagnostics and treatment strategies. Electronic health records (EHRs), particularly those from intensive care units (ICUs), contain rich temporal clinical data that can be leveraged for this purpose. In this work, we introduce ICU-TSB (Temporal Stratification Benchmark), the first comprehensive benchmark for evaluating patient stratification based on temporal patient representation learning using three publicly available ICU EHR datasets. A key contribution of our benchmark is a novel hierarchical evaluation framework utilizing disease taxonomies to measure the alignment of discovered clusters with clinically validated disease groupings. In our experiments with ICU-TSB, we compared statistical methods and several recurrent neural networks, including LSTM and GRU, for their ability to generate effective patient representations for subsequent clustering of patient trajectories. Our results demonstrate that temporal representation learning can rediscover clinically meaningful patient cohorts; nevertheless, it remains a challenging task, with v-measuring varying from up to 0.46 at the top level of the taxonomy to up to 0.40 at the lowest level. To further enhance the practical utility of our findings, we also evaluate multiple strategies for assigning interpretable labels to the identified clusters. The experiments and benchmark are fully reproducible and available at https://github.com/ds4dh/CBMS2025stratification.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Akiba, T.; Sano, S.; Yanase, T.; Ohta, T.; and Koyama, M. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In ACM SIGKDD Int. Conference on Knowledge Discovery & Data Mining, 2623–2631. ACM
work page 2019
-
[4]
Bennett, N.; Ple c ko, D.; Ukor, I.-F.; Meinshausen, N.; and B \"u hlmann, P. 2023. ricu: R’s interface to intensive care data. GigaScience, 12: giad041
work page 2023
-
[5]
Bornet, A.; Proios, D.; Yazdani, A.; Jaume-Santero, F.; Haller, G.; Choi, E.; and Teodoro, D. 2025. Comparing neural language models for medical concept representation and patient trajectory prediction. Artificial Intelligence in Medicine
work page 2025
-
[6]
S.; Gibbs, C.; Martin, S.; Firman, T.; Gaskell, A.; Fosdick, B.; and Layer, R
Bradshaw, M. S.; Gibbs, C.; Martin, S.; Firman, T.; Gaskell, A.; Fosdick, B.; and Layer, R. 2024. Hypothesis generation for rare and undiagnosed diseases through clustering and classifying time-versioned biological ontologies. PloS one
work page 2024
-
[7]
Carr, O.; Javer, A.; Rockenschaub, P.; Parsons, O.; and Durichen, R. 2021. Longitudinal patient stratification of electronic health records with flexible adjustment for clinical outcomes. In Machine Learning for Health, 220--238
work page 2021
-
[8]
W.; Flores, G.; Xue, Y.; and Dai, A
Choi, E.; Xu, Z.; Li, Y.; Dusenberry, M. W.; Flores, G.; Xue, Y.; and Dai, A. M. 2020. Learning the Graphical Structure of Electronic Health Records with Graph Convolutional Transformer . ArXiv:1906.04716 [cs, stat]
arXiv 2020
Show all 23 references
-
[9]
C.; Ver Steeg, G.; and Galstyan, A
Harutyunyan, H.; Khachatrian, H.; Kale, D. C.; Ver Steeg, G.; and Galstyan, A. 2019. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1): 96. Publisher: Nature Publishing Group
2019
-
[10]
Jaume-Santero, F.; Zhang, B.; Proios, D.; Yazdani, A.; Gouareb, R.; Bjelogrlic, M.; and Teodoro, D. 2022. Cluster analysis of low-dimensional medical concept representations from electronic health records. In International conference on health information science, 313--324
2022
-
[11]
E.; Bulgarelli, L.; Shen, L.; Gayles, A.; Shammout, A.; Horng, S.; Pollard, T
Johnson, A. E.; Bulgarelli, L.; Shen, L.; Gayles, A.; Shammout, A.; Horng, S.; Pollard, T. J.; Hao, S.; Moody, B.; Gow, B.; et al. 2023. MIMIC-IV, a freely accessible electronic health record dataset. Scientific data, 10(1): 1
2023
-
[12]
E.; Smith, K.; Chiou, J.; Cole, J
Kim, H.; Westerman, K. E.; Smith, K.; Chiou, J.; Cole, J. B.; and Majarian, T. 2023. High-throughput genetic clustering of type 2 diabetes loci reveals heterogeneous mechanistic pathways of metabolic disease. Diabetologia, 66: 495--507
2023
-
[13]
Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. arXiv:1711.05101
2019 arXiv
-
[14]
Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. 2011. Scikit-learn: Machine learning in Python. Journal of machine learning research, 12(Oct)
2011
-
[15]
J.; Johnson, A
Pollard, T. J.; Johnson, A. E. W.; Raffa, J. D.; Celi, L. A.; Mark, R. G.; and Badawi, O. 2018. The eICU Collaborative Research Database , a freely available multi-center database for critical care research. Scientific Data, 5(1): 180178
2018
-
[16]
Proios, D.; Yazdani, A.; Bornet, A.; Ehrsam, J.; Rekik, I.; and Teodoro, D. 2023. Leveraging patient similarities via graph neural networks to predict phenotypes from temporal data. In 2023 IEEE 10th International Conference on Data Science and Advanced Analytics (DSAA), 1--10
2023
-
[17]
Rosenberg, A.; and Hirschberg, J. 2007. V-measure: A conditional entropy-based external cluster evaluation measure. Journal of Computational Linguistics
2007
-
[18]
Sadeghi, S.; Hempel, L.; Rodemund, N.; and Kirsten, T. 2024. Salzburg Intensive Care database ( SICdb ): a detailed exploration and comparative analysis with MIMIC - IV . Scientific Reports. Publisher: Nature Publishing Group
2024
-
[19]
E.; Naderi, A.; Barbosa-Morais, N
Teschendorff, A. E.; Naderi, A.; Barbosa-Morais, N. L.; and Caldas, C. 2006. PACK : Profile Analysis using Clustering and Kurtosis to find molecular classifiers in cancer. Bioinformatics, 22(18): 2269--2275
2006
-
[20]
van de Water, R.; Schmidt, H.; Elbers, P.; Thoral, P.; Arnrich, B.; and Rockenschaub, P. 2024. Yet Another ICU Benchmark : A Flexible Multi - Center Framework for Clinical ML . ArXiv:2306.05109 [cs]
2024 arXiv
-
[21]
X.; Epps, J.; and Bailey, J
Vinh, N. X.; Epps, J.; and Bailey, J. 2010. Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. Journal of Machine Learning Research
2010
-
[22]
A.; Roger, M.; and Brenner, B
Vrbik, I.; Stephens, D. A.; Roger, M.; and Brenner, B. G. 2015. The Gap Procedure : for the identification of phylogenetic clusters in HIV -1 sequence data. BMC Bioinformatics
2015
-
[23]
Zang, C.; and Wang, F. 2021. SCEHR: Supervised Contrastive Learning for Clinical Risk Prediction using Electronic Health Records. CoRR, abs/2110.04943
2021 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.