REVIEW 2 major objections 5 minor
CoMedBench: A Multi-Source Benchmark of Synthetic Medical Data Fidelity and Downstream Utility
T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read CoMedBench, a 37-task benchmark of synthetic medical data, finds that synthetic training data preserves most predictive signal on static tabular tasks but loses much of it on temporal ICU tasks, with AUPRC utility falling to 64% for the…
desk verdict Useful controlled benchmark for tabular synthetic medical data, but the temporal-bottleneck claim is untested because the ICU tasks collapse time series into static summaries. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the utility ratio, Utility% = AUCTSTR/AUCTRTR × 100, where TRTR trains and tests on real data and TSTR trains on synthetic data and tests on the same held-out real set. To make every task comparable, CoMedBench reduces each dataset to a single flat table: ICU stays are summarized per variable into first, min, max, mean (and where available last, median, std) plus demographic features, and four single-table generators (CTGAN, TVAE, CopulaGAN, GaussianCopula) are fit under a common clinical-validity layer. Fidelity is scored as Overall = (Shapes + Trends)/2, where Shapes averages marginal KS/TV matches and Trends averages pairwise correlation matches. This design isolates the generator as the only variable between runs, so performance gaps can be attributed to synthesis rather than preprocessing or evaluation differences.
What would settle it
Run the benchmark's TRTR protocol on real ICU stays twice: once on the paper's per-stay summary features and once on the raw multivariate time series with a sequence model such as a GRU-D or transformer. If the sequence model materially beats the summary model on real training data for mortality or readmission, the summary representation itself discards task-relevant temporal signal and the reported time-series utility gaps cannot be attributed to the synthesizers; if the two match, the paper's interpretation survives.
Extended reading notes
Core claim
The paper's central claim is that synthetic clinical data is already a credible substitute for real data in static tabular prototyping and benchmarking, but not yet for temporal, highly imbalanced ICU prediction. Under a shared training and evaluation engine, utility measured as the ratio of synthetic-trained to real-trained AUROC on the same held-out real test set reaches 90.6% for CoMed-CTGAN and 97.3% for CoMed-TVAE on tabular tasks, while on temporal ICU tasks CoMed-CTGAN falls to 81.6% AUROC and 64.0% AUPRC; several ICU mortality tasks retain only 30–45% of real-data AUPRC. The paper further claims that no single generator dominates, that classifier choice matters little relative to data modality, and that overall fidelity correlates only moderately with utility (r=0.67, rho=0.74 for CoMed-CTGAN), making fidelity a necessary but insufficient proxy. It concludes that downstream utility must be evaluated directly, and that rare-event imbalance is the main bottleneck.
Load-bearing premise
The load-bearing premise is that summarizing each ICU stay into per-stay statistics (first, min, max, mean, and sometimes last, median, std) preserves the signal needed for the downstream tasks, so real-versus-synthetic gaps can be blamed on the generator rather than on information lost in the summarization. If this premise fails, the paper's time-series conclusions become conclusions about tabular summaries, not about temporal synthesis.
Editorial extensions
If this is right
- Static tabular clinical prediction can be prototyped and benchmarked on synthetic data without much loss: mean AUROC utility is about 91% for the reference CTGAN and 97% for TVAE, with some datasets meeting the real-data baseline.
- Rare-event ICU tasks are the clearest failure mode: AUPRC utility for CTGAN falls to about 64% on average and to 30–45% for rare mortality tasks, so reporting AUROC alone overstates the usefulness of synthetic temporal data.
- No single generator is a safe default; the TVAE is the most faithful and strongest overall, while the Gaussian-copula model is competitive on small or highly imbalanced cohorts, so generator choice should be task-dependent.
- Statistical fidelity—especially correlation structure—is a useful screen but not a certificate: the overall fidelity–utility correlation is r=0.67 for CTGAN and only r=0.40 for TVAE, so utility needs its own measurement.
- The data modality, not the downstream classifier, drives utility; the same five classifiers produce similar relative results across generators.
Reading between the lines
- A direct test of the paper's time-series interpretation is to replace the per-stay summaries with a sequence-native generator and raw ICU trajectories; since the paper explicitly drops fine-grained dynamics, the observed ICU gap could shrink or grow under sequence-aware synthesis.
- The utility ratio could be operationalized as a release gate in a synthetic-data pipeline: generate candidate tables, test on held-out real data, and refuse releases whose TSTR/TRTR retention falls below a clinical-task threshold, because fidelity alone cannot certify usefulness.
- The benchmark's near-ceiling utility for TVAE suggests the fidelity–utility correlation is not a universal constant but depends on generator quality; future evaluations should report per-generator correlations rather than a single headline number.
- Requiring an imbalance-sensitive metric like AUPRC in every medical synthetic-data evaluation would sharpen the field's comparisons, since most clinical prediction targets are rare events.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CoMedBench, a benchmark for evaluating synthetic medical data across 37 dataset-task pairs: 20 static tabular tasks and 17 ICU time-series tasks, the latter represented as per-stay summary feature vectors. The benchmark applies a common clinical-validity layer, four synthetic generators (CTGAN, TVAE, CopulaGAN, GaussianCopula), statistical fidelity metrics, and TRTR/TSTR downstream utility measured with five classifiers on a shared held-out real test set. The headline results are that synthetic training data preserves most tabular AUROC utility (90.6% for CoMed-CTGAN, 97.3% for CoMed-TVAE), while ICU tasks are harder and more generator-sensitive (81.6% AUROC and 64.0% AUPRC for CoMed-CTGAN), and that Overall statistical fidelity correlates only moderately with utility (Pearson r=0.67, Spearman rho=0.74 for CoMed-CTGAN). The paper concludes that synthetic data is a credible substitute for static tabular prototyping, with rare-event temporal ICU tasks as the main remaining bottleneck.
Significance. If the reported numbers hold, CoMedBench is a useful controlled resource for the synthetic-medical-data community. Its strengths include the use of an external held-out real test set for TSTR evaluation, a consistent evaluation engine across a broad set of public datasets and generators, per-classifier and per-dataset results in the supplement, and a direct analysis of the fidelity-utility relationship. The finding that AUROC can substantially overstate synthetic-data utility for rare outcomes is practically important. The main caveat is that the temporal-bottleneck interpretation is not supported by the experimental design, because the ICU tasks are evaluated on static per-stay summaries rather than on the original time series; the value of the benchmark as a static-table benchmark is nonetheless substantial.
major comments (2)
- [Methodology (Datasets, Tasks, and Representation; Figure 3) and Conclusion] The paper's central claim that 'temporal structure remains a bottleneck' is not supported by the experimental design. Every ICU stay is collapsed into a per-stay feature vector using {v_first, v_min, v_max, v_mean, and optionally v_last, v_median, v_std} before any synthesis or evaluation, and the Conclusion explicitly states that 'claims about fine-grained temporal dynamics are out of scope.' The observed utility gap in Table 3 (e.g., CoMed-CTGAN 81.6% AUROC and 64.0% AUPRC on the time-series block) could therefore reflect information loss in the summarization, the higher column counts of the ICU tables (109-168 columns in Supplement Tables 1 and 2), or rare-event imbalance, rather than any difficulty in temporal synthesis. Please either evaluate sequence-native generators on the raw series or re-scope the claims to 'static per-stay summaries of ICU data' and remove the temporal-bottleneck phrasing from the abstract and conclusion.
- [Supplement Section 7 (Implementation Details)] All experiments use a single synthetic sample and a single stratified 80/20 train/test split with random_state=42. The headline point estimates (mean AUROC utility 90.6%, 97.3%, 81.6%; AUPRC 64.0%) and the fidelity-utility correlations (r=0.67, rho=0.74) are therefore single-replication quantities with no measure of variability across synthetic draws or data splits. Since the benchmark's purpose includes comparing generators, please add multiple synthetic samples and repeated splits, or otherwise quantify sensitivity, before treating the generator ranking and the quantitative utility gaps as stable benchmark findings.
minor comments (5)
- [Abstract] The abstract contains grammatical and typographical issues, including '20 static tabular and 17 temporal ICU time-series-drawn from seven public data sources' and 'consists of'; these should be rewritten for clarity.
- [Methodology, Eqs. (9)-(11) and Figure 4] The Data Validity metric is computed against boundary and category constraints that the clinical-validity layer itself enforces during generation, so the near-universal 1.000 Data Validity scores in Supplement Tables 3-4 are expected by construction. Please add a sentence stating that Data Validity is a conformity check rather than an independent statistical-fidelity measure.
- [Supplement Tables 5-8] The per-classifier tables include a 'Paper' column whose entries are mostly '–'; please remove this column or clarify what it is intended to report.
- [Table 3] Some rows in the Real (TRTR) column are missing (e.g., several MIMIC-III rows), while others are present; please make explicit which baselines were re-computed and which were taken from the published benchmarks cited in the Supplement.
- [Methodology (Evaluation and Supplement Section 7)] The downstream classifiers use different hyperparameters for tabular and time-series tasks (e.g., Random Forest n_estimators 300 vs. 400, MLP architectures (100,) vs. (128,64)). Since the paper claims that 'only the synthesizer varies,' it would be clearer to state that this invariance holds within each modality, not across modalities.
Circularity Check
Utility numbers are externally anchored, but the fidelity diagnostic includes a by-construction Data Validity score, and the 'temporal bottleneck' conclusion is drawn from a pipeline that flattens time series into static per-stay rows.
-
self definitional
[Methodology, Clinical Validity Controls and Evaluation (Eq. 9); Supplementary Section 7 (generator settings)]
"Data Validity averages a per-column adherence score: boundary adherence for numerical columns and category adherence for categorical columns, BA(c) = 1/|S| Σ_{s∈Sc} 1[min_c^R ≤ s ≤ max_c^R] ... The layer uses schema information and task-specific validity rules ... domain-validity constraints applied during fitting; enforce_min_max_values=True ... High-cardinality categorical fields are encoded so that generators emit only categories observed in training."
The clinical-validity layer constrains generation to the real data's min/max ranges and category sets, and the Data Validity score BA(c)/CA(c) measures exactly whether synthetic values fall inside those same real-determined boundaries/categories. Adherence is therefore guaranteed by the fitting procedure rather than being an independent property of the generator; the reported near-1.000 Data Validity values (Tables 3/4) are an audit of the constraint, not evidence of fidelity. The central utility results do not depend on this score, but the diagnostic fidelity report presents it as a measured outcome.
-
renaming known result
[Methodology (Datasets, Tasks, and Representation); Abstract; Conclusion]
"each ICU stay's multivariate time series is summarized into a per-stay feature vector: each clinical variable v is represented by vfirst, vmin, vmax, and vmean ... yielding one fixed-width row per stay ... Because our pipeline summarizes each ICU stay into a per-stay feature vector, claims about fine-grained temporal dynamics are out of scope ... Temporal ICU tasks are harder and more generator-sensitive ... temporal structure remains a bottleneck."
The 'temporal' arm of the benchmark does not evaluate temporal structure: ICU series are collapsed to {first, min, max, mean, ...} per-stay features and then passed through the same single-table generators and flat classifiers as the tabular arm. No ordering, irregular sampling, missingness pattern, or dynamics is modeled or scored. Attributing the lower utility on these tasks to 'temporal structure' therefore uses the dataset-origin label as the explanation; the observed gap could equally be driven by the much larger column counts and rare-event imbalance of these static tables. The conclusion's own scope disclaimer concedes that fine-grained temporal dynamics were not tested, contradicting the abstract's causal phrasing.
full rationale
The paper's central utility evaluation is not circular: TSTR models trained on synthetic data are tested on a held-out real test set and compared to TRTR on the same set (Eq. 12), so the headline AUROC/AUPRC utility ratios are anchored to external real labels. The fidelity-to-utility correlation is also computed from independent axes. The circular components are confined to (a) the Data Validity diagnostic, which scores adherence to constraints that were used to generate the data, and (b) the temporal-bottleneck interpretation, which is asserted for tasks that the pipeline itself flattened into static per-stay tables. The first is a minor auxiliary metric; the second affects a central headline but is an interpretive overreach rather than a fitted parameter renamed as a prediction. The self-citations in the paper (Pias et al. 2025, 2026) are background references and do not carry the derivation. Score 4 reflects partial circularity in diagnostics and interpretation while the core benchmark numbers remain independently grounded.
Assumptions & free parameters
free parameters (4)
- random_seed =
42
- per-dataset generator epochs and batch size =
300 to 2000 epochs, batch size 50 to 5000 depending on dataset
- generator architecture hyperparameters =
embedding_dim 128 or 512, generator/discriminator dims (256,256) or (256,512,256), pac=10
- fidelity Overall weights =
equal weights 0.5 on Shapes and 0.5 on Trends
assumptions (4)
- domain assumption Per-stay summary statistics capture the clinically relevant signal for ICU prediction tasks
- domain assumption TSTR and TRTR with one 80/20 split and five fixed classifiers is a valid measure of downstream utility
- domain assumption The SDV implementations of CTGAN, TVAE, CopulaGAN, and GaussianCopula are representative of their generator families
- ad hoc to paper Clinical-validity constraints derived from real training data (min/max ranges, allowed categories) are appropriate to impose on synthetic data
Cite this review
Pith. "Pith review of CoMedBench: A Multi-Source Benchmark of Synthetic Medical Data Fidelity and Downstream Utility." pith.science (2026). https://pith.science/paper/OSIOSC76
@misc{pith2026260812805,
author = {Pith},
title = {Pith review of: CoMedBench: A Multi-Source Benchmark of Synthetic Medical Data Fidelity and Downstream Utility},
year = {2026},
howpublished = {\url{https://pith.science/paper/OSIOSC76}},
note = {Machine review of arXiv:2608.12805}
}
read the original abstract
Access to clinical data is essential for developing reliable healthcare machine learning systems, but direct use of electronic health records is constrained by privacy regulation, institutional review, data-use agreements, and the risk of re-identification. Synthetic data promises a practical alternative: it can preserve useful statistical and clinical structure while reducing exposure of sensitive patient records. Prior studies often evaluate a single generator, one dataset, or a narrow downstream task, making it difficult to know when synthetic data can support model development and when it fails to preserve task-critical signal. We introduce CoMedBench, a reproducible benchmark that evaluates a family of generators under a common clinical-validity framework and one shared training and evaluation engine, spanning static tabular and temporal downstream tasks on established critical-care datasets. In total the benchmark spans 37 dataset-task pairs across two modalities consists of 20 static tabular and 17 temporal ICU time-series-drawn from seven public data sources: three intensive-care databases (MIMIC-III, MIMIC-IV, and eICU) together with the UCI Machine Learning Repository, the CDC BRFSS diabetes cohort (2015), NHANES (1999-2014), and the pycox survival datasets (GBSG and METABRIC). The benchmark evaluates both statistical fidelity and task utility by comparing models trained and tested across real and synthetic data. In these settings, synthetic training data preserves most of the downstream signal: on tabular tasks the reference generator CoMed-CTGAN retains a mean AUROC utility (the synthetic-to-real performance ratio) of 90.6%, rising to 97.3% for the strongest generator, CoMed-TVAE. Temporal ICU tasks are harder and more generator-sensitive: CoMed-CTGAN retains 81.6% (AUROC) and only 64.0% under the imbalance-sensitive AUPRC, whereas CoMed-TVAE still retains ~95% (AUROC).
Figures
Figures from the paper (2 more)
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.