{"id":"4e0c7666-3a6a-48ab-bb48-82e6c48d7209","arxiv_id":"2608.12805","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CoMedBench evaluates four synthetic-data generators on 37 clinical prediction tasks, finding tabular AUROC utility of 90.6% for CoMed-CTGAN and 97.3% for CoMed-TVAE, with ICU time-series utility falling to 81.6% and to 64.0% under AUPRC.","lead":"CoMedBench tests four synthetic medical-data generators across 37 prediction tasks and finds that synthetic data preserves most signal in static tabular tasks but loses more on summarized ICU time-series tasks, especially for rare outcomes. A broad, controlled benchmark matters for healthcare teams deciding when synthetic patient data can safely replace real records in model development.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The temporal-bottleneck conclusion is not tested: ICU time series are collapsed to per-stay summary features, so the time-series utility gap may reflect summarization loss, not temporal synthesis failure.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing weakness: the per-stay summarization means that the ICU tasks are not actually temporal in the evaluation pipeline, yet the abstract and conclusion frame the utility drop as evidence about temporal structure. This concern is supported by the manuscript itself: the Conclusion disclaims fine-grained temporal dynamics, and Figure 3 shows the summarization step. The paper's tables and numbers are not internally contradicted, so the tabular headline (90.6% and 97.3% AUROC utility) likely stands; however, the temporal conclusion is an overreach relative to what was measured. A concrete sequence-native TRTR/TSTR comparison on the raw ICU series would settle whether the bottleneck is temporal synthesis, summarization, or simply high-dimensional tabular generation under imbalance. The verdict should remain CONDITIONAL: the authors should either rephrase the temporal claims to match the static-summary design, add sequence-native experiments, or explicitly limit the time-series conclusion to the summarized representation. The reader's other points (multi-seed variability, code release, validity-layer trade-off) are secondary but reinforce the conditional verdict.","tokens_in":45170,"tokens_out":2644,"duration_ms":30231,"concrete_test":"Run the MIMIC-III and MIMIC-IV ICU mortality and readmission tasks with a sequence-native model (e.g., GRU-D or LSTM) on the raw multivariate time series, using the same held-out real test set and the same TRTR/TSTR protocol. If TRTR AUROC/AUPRC on raw sequences is materially higher (e.g., >5 points) than TRTR on the per-stay summary features, the summarization is lossy and the temporal bottleneck claim is confounded. Additionally, if feasible, compare TSTR with a sequence-native generator (e.g., TimeGAN or a conditional RNN) trained on raw sequences; if such a generator retains substantially more signal than CoMed-CTGAN on summary features, the difference is attributable to representation and generator choice rather than to temporal structure per se.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that temporal ICU tasks are harder and that \"temporal structure remains a bottleneck\" is not directly supported by the experimental design. In Methodology (Figure 3), every ICU stay's multivariate time series is reduced to a per-stay feature vector using {first, min, max, mean, and optionally last, median, std} along with demographic and contextual features. All fidelity scores and downstream utility metrics are then computed on this flat, static table. The Conclusion explicitly states: \"Because our pipeline summarizes each ICU stay into a per-stay feature vector, claims about fine-grained temporal dynamics are out of scope.\" This is an internal contradiction: the abstract and conclusion attribute the lower AUROC/AUPRC on ICU tasks to temporal structure, yet the pipeline never models or evaluates temporal dynamics. The gap could instead reflect the difficulty of generating high-dimensional flat tables with many columns (e.g., 109-168 features on MIMIC tasks) combined with rare-event imbalance; column count and imbalance differ substantially between the \"tabular\" and \"time-series\" task sets. If the per-stay summarization discards information that sequence models would exploit (trends, missingness patterns, irregular timing), then the observed utility gap is a statement about synthesizing lossy static features, not about temporal synthesis. This makes the headline conclusion about temporal bottlenecks vulnerable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoMedBench, a benchmark for evaluating synthetic medical data across 37 dataset-task pairs: 20 static tabular tasks and 17 ICU time-series tasks, the latter represented as per-stay summary feature vectors. The benchmark applies a common clinical-validity layer, four synthetic generators (CTGAN, TVAE, CopulaGAN, GaussianCopula), statistical fidelity metrics, and TRTR/TSTR downstream utility measured with five classifiers on a shared held-out real test set. The headline results are that synthetic training data preserves most tabular AUROC utility (90.6% for CoMed-CTGAN, 97.3% for CoMed-TVAE), while ICU tasks are harder and more generator-sensitive (81.6% AUROC and 64.0% AUPRC for CoMed-CTGAN), and that Overall statistical fidelity correlates only moderately with utility (Pearson r=0.67, Spearman rho=0.74 for CoMed-CTGAN). The paper concludes that synthetic data is a credible substitute for static tabular prototyping, with rare-event temporal ICU tasks as the main remaining bottleneck.","tokens_in":45407,"tokens_out":5541,"duration_ms":56802,"significance":"If the reported numbers hold, CoMedBench is a useful controlled resource for the synthetic-medical-data community. Its strengths include the use of an external held-out real test set for TSTR evaluation, a consistent evaluation engine across a broad set of public datasets and generators, per-classifier and per-dataset results in the supplement, and a direct analysis of the fidelity-utility relationship. The finding that AUROC can substantially overstate synthetic-data utility for rare outcomes is practically important. The main caveat is that the temporal-bottleneck interpretation is not supported by the experimental design, because the ICU tasks are evaluated on static per-stay summaries rather than on the original time series; the value of the benchmark as a static-table benchmark is nonetheless substantial.","major_comments":[{"comment":"The paper's central claim that 'temporal structure remains a bottleneck' is not supported by the experimental design. Every ICU stay is collapsed into a per-stay feature vector using {v_first, v_min, v_max, v_mean, and optionally v_last, v_median, v_std} before any synthesis or evaluation, and the Conclusion explicitly states that 'claims about fine-grained temporal dynamics are out of scope.' The observed utility gap in Table 3 (e.g., CoMed-CTGAN 81.6% AUROC and 64.0% AUPRC on the time-series block) could therefore reflect information loss in the summarization, the higher column counts of the ICU tables (109-168 columns in Supplement Tables 1 and 2), or rare-event imbalance, rather than any difficulty in temporal synthesis. Please either evaluate sequence-native generators on the raw series or re-scope the claims to 'static per-stay summaries of ICU data' and remove the temporal-bottleneck phrasing from the abstract and conclusion.","section":"Methodology (Datasets, Tasks, and Representation; Figure 3) and Conclusion"},{"comment":"All experiments use a single synthetic sample and a single stratified 80/20 train/test split with random_state=42. The headline point estimates (mean AUROC utility 90.6%, 97.3%, 81.6%; AUPRC 64.0%) and the fidelity-utility correlations (r=0.67, rho=0.74) are therefore single-replication quantities with no measure of variability across synthetic draws or data splits. Since the benchmark's purpose includes comparing generators, please add multiple synthetic samples and repeated splits, or otherwise quantify sensitivity, before treating the generator ranking and the quantitative utility gaps as stable benchmark findings.","section":"Supplement Section 7 (Implementation Details)"}],"minor_comments":[{"comment":"The abstract contains grammatical and typographical issues, including '20 static tabular and 17 temporal ICU time-series-drawn from seven public data sources' and 'consists of'; these should be rewritten for clarity.","section":"Abstract"},{"comment":"The Data Validity metric is computed against boundary and category constraints that the clinical-validity layer itself enforces during generation, so the near-universal 1.000 Data Validity scores in Supplement Tables 3-4 are expected by construction. Please add a sentence stating that Data Validity is a conformity check rather than an independent statistical-fidelity measure.","section":"Methodology, Eqs. (9)-(11) and Figure 4"},{"comment":"The per-classifier tables include a 'Paper' column whose entries are mostly '–'; please remove this column or clarify what it is intended to report.","section":"Supplement Tables 5-8"},{"comment":"Some rows in the Real (TRTR) column are missing (e.g., several MIMIC-III rows), while others are present; please make explicit which baselines were re-computed and which were taken from the published benchmarks cited in the Supplement.","section":"Table 3"},{"comment":"The downstream classifiers use different hyperparameters for tabular and time-series tasks (e.g., Random Forest n_estimators 300 vs. 400, MLP architectures (100,) vs. (128,64)). Since the paper claims that 'only the synthesizer varies,' it would be clearer to state that this invariance holds within each modality, not across modalities.","section":"Methodology (Evaluation and Supplement Section 7)"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The benchmark resource is genuinely useful and the TSTR evaluation is anchored to external real test sets, which is a clear strength. The main obstacle is the unsupported temporal-bottleneck interpretation; this can be fixed by re-scoping the claims to flat per-stay ICU benchmarks and by adding multi-seed and multi-synthetic-sample robustness. I do not recommend rejection because the underlying benchmark and its per-classifier data would remain valuable after such a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful benchmark for synthetic tabular medical data, but its marquee finding about 'temporal ICU tasks' is not actually tested. The pipeline reduces each ICU stay to per-stay summary statistics (first, min, max, mean, etc.), so the 'temporal' utility gap is really about synthesizing high-dimensional flat tables, not about time-series generation. The authors even admit this in the conclusion, which makes the abstract read as overreach.\n\nWhat's new: CoMedBench applies one preprocessing and evaluation engine to four generators (CTGAN, TVAE, CopulaGAN, GaussianCopula) across 20 tabular and 17 ICU task pairs from seven public sources. That is a bigger controlled comparison than anything I know of in the synthetic clinical data literature. The descriptive numbers look internally consistent: tabular AUROC utility around 90-97% for CTGAN and TVAE, and plausible degradation on the imbalanced ICU summary tasks. The supplementary material is unusually complete, with per-classifier results for every dataset-task pair and an ablation of the clinical-validity layer. That is reproducible evidence, if the code ever ships.\n\nSoft spots, in order of importance. First, the temporal conclusion. The 'time-series' tasks are static per-stay features; the column count and imbalance differ between the two modality groups, so the gap could be driven by high dimensionality and rare outcomes rather than temporal dynamics. Fix: rename the axis, or add a sequence-native generator to actually test the claim. Second, the entire benchmark runs one synthetic sample on one 80/20 split with one seed. That is fine for a first pass, but it means the utility ratios have no error bars. Third, no code is released; 'reproducible benchmark' is too strong without it. Fourth, the ablation shows that disabling the clinical-validity layer often improves downstream AUROC (e.g., MIMIC-III heart failure mortality goes from 0.480 to 0.637 for CTGAN), and the main text ignores this trade-off. That is worth a paragraph.\n\nThe fidelity-utility correlation analysis is interesting but secondary. The Overall r=0.67 for CTGAN is moderate; the main text's 'necessary but insufficient proxy' is a fair reading. Dataset size shows no significant correlation, which is a useful negative result.\n\nBottom line: the tabular claims are solid and worth taking seriously. The temporal claim is overstated, and the benchmark lacks the reproducibility details needed for full credit. I would send this to a serious referee, but I would expect the authors to release code, add uncertainty, and redraw the temporal conclusion before publication. If they do, I would cite it.","headline":"Useful controlled benchmark for tabular synthetic medical data, but the temporal-bottleneck claim is untested because the ICU tasks collapse time series into static summaries.","tokens_in":45969,"tokens_out":2440,"would_cite":true,"duration_ms":24664,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoMedBench, a 37-task benchmark of synthetic medical data, finds that synthetic training data preserves most predictive signal on static tabular tasks but loses much of it on temporal ICU tasks, with AUPRC utility falling to 64% for the…","keywords":["synthetic medical data","benchmarking","downstream utility","statistical fidelity","ICU time series","tabular data","electronic health records","TSTR evaluation"],"falsifier":"Run the benchmark's TRTR protocol on real ICU stays twice: once on the paper's per-stay summary features and once on the raw multivariate time series with a sequence model such as a GRU-D or transformer. If the sequence model materially beats the summary model on real training data for mortality or readmission, the summary representation itself discards task-relevant temporal signal and the reported time-series utility gaps cannot be attributed to the synthesizers; if the two match, the paper's interpretation survives.","tokens_in":44946,"feed_emoji":"🏥","tokens_out":7437,"duration_ms":70269,"temperature":0.7,"pith_summary":"CoMedBench asks a practical question: when can synthetic medical data replace real patient records for training predictive models? It answers with a controlled comparison across 37 dataset-task pairs, using a single evaluation engine so that only the generator changes between runs. On static tabular tasks, synthetic training data preserves most of the real-data signal, with mean AUROC utility of 90.6% for the reference generator CoMed-CTGAN and 97.3% for the strongest generator CoMed-TVAE. On temporal ICU tasks the picture is worse and more generator-dependent: CoMed-CTGAN retains 81.6% of AUROC but only 64.0% of AUPRC, and rare-outcome ICU mortality tasks retain roughly 30–45% of AUPRC. The paper also finds that statistical fidelity is only a moderate predictor of utility, so usefulness must be measured directly.","feed_headline":"Synthetic tables hit 97% of real-data utility, ICU 64%","feed_subtitle":"A 37-task benchmark shows synthetic EHRs can replace real records for static prediction, but rare-event ICU tasks still suffer.","key_machinery":"The load-bearing object is the utility ratio, Utility% = AUCTSTR/AUCTRTR × 100, where TRTR trains and tests on real data and TSTR trains on synthetic data and tests on the same held-out real set. To make every task comparable, CoMedBench reduces each dataset to a single flat table: ICU stays are summarized per variable into first, min, max, mean (and where available last, median, std) plus demographic features, and four single-table generators (CTGAN, TVAE, CopulaGAN, GaussianCopula) are fit under a common clinical-validity layer. Fidelity is scored as Overall = (Shapes + Trends)/2, where Shapes averages marginal KS/TV matches and Trends averages pairwise correlation matches. This design isolates the generator as the only variable between runs, so performance gaps can be attributed to synthesis rather than preprocessing or evaluation differences.","core_discovery":"The paper's central claim is that synthetic clinical data is already a credible substitute for real data in static tabular prototyping and benchmarking, but not yet for temporal, highly imbalanced ICU prediction. Under a shared training and evaluation engine, utility measured as the ratio of synthetic-trained to real-trained AUROC on the same held-out real test set reaches 90.6% for CoMed-CTGAN and 97.3% for CoMed-TVAE on tabular tasks, while on temporal ICU tasks CoMed-CTGAN falls to 81.6% AUROC and 64.0% AUPRC; several ICU mortality tasks retain only 30–45% of real-data AUPRC. The paper further claims that no single generator dominates, that classifier choice matters little relative to data modality, and that overall fidelity correlates only moderately with utility (r=0.67, rho=0.74 for CoMed-CTGAN), making fidelity a necessary but insufficient proxy. It concludes that downstream utility must be evaluated directly, and that rare-event imbalance is the main bottleneck.","pith_inferences":["A direct test of the paper's time-series interpretation is to replace the per-stay summaries with a sequence-native generator and raw ICU trajectories; since the paper explicitly drops fine-grained dynamics, the observed ICU gap could shrink or grow under sequence-aware synthesis.","The utility ratio could be operationalized as a release gate in a synthetic-data pipeline: generate candidate tables, test on held-out real data, and refuse releases whose TSTR/TRTR retention falls below a clinical-task threshold, because fidelity alone cannot certify usefulness.","The benchmark's near-ceiling utility for TVAE suggests the fidelity–utility correlation is not a universal constant but depends on generator quality; future evaluations should report per-generator correlations rather than a single headline number.","Requiring an imbalance-sensitive metric like AUPRC in every medical synthetic-data evaluation would sharpen the field's comparisons, since most clinical prediction targets are rare events."],"forward_implications":["Static tabular clinical prediction can be prototyped and benchmarked on synthetic data without much loss: mean AUROC utility is about 91% for the reference CTGAN and 97% for TVAE, with some datasets meeting the real-data baseline.","Rare-event ICU tasks are the clearest failure mode: AUPRC utility for CTGAN falls to about 64% on average and to 30–45% for rare mortality tasks, so reporting AUROC alone overstates the usefulness of synthetic temporal data.","No single generator is a safe default; the TVAE is the most faithful and strongest overall, while the Gaussian-copula model is competitive on small or highly imbalanced cohorts, so generator choice should be task-dependent.","Statistical fidelity—especially correlation structure—is a useful screen but not a certificate: the overall fidelity–utility correlation is r=0.67 for CTGAN and only r=0.40 for TVAE, so utility needs its own measurement.","The data modality, not the downstream classifier, drives utility; the same five classifiers produce similar relative results across generators."],"supporting_citations":[{"why":"Supplies the MIMIC-III and MIMIC-IV intensive-care databases used for 12 of the 17 temporal ICU dataset-task pairs.","marker":"Johnson et al. 2016, 2023"},{"why":"Supplies the eICU multi-center critical-care database used for the remaining ICU tasks.","marker":"Pollard et al. 2018"},{"why":"Defines the CTGAN and TVAE generator families that become CoMed-CTGAN and CoMed-TVAE.","marker":"Xu et al. 2019"},{"why":"Provides the single-table synthesizer implementations and the core fidelity metrics the benchmark adapts.","marker":"Patki, Wedge, and Veeramachaneni 2016"},{"why":"Establishes the downstream ICU prediction tasks (mortality, readmission, length-of-stay) that define the benchmark's utility evaluation.","marker":"Harutyunyan et al. 2019"},{"why":"Motivates why temporal patterns and missingness in ICU data carry predictive signal that a summary representation may not preserve.","marker":"Che et al. 2018"}],"fun_headline_variants":["Synthetic tables hit 97% real utility, ICU drops to 64%","CoMedBench: 97% tabular utility, 64% ICU AUPRC","Synthetic data: 97% on tables, 64% on ICU events","ICU imbalance crushes synthetic data: 64% vs 97% on tables","Synthetic EHRs: near-real for tables, 64% for ICU events"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that summarizing each ICU stay into per-stay statistics (first, min, max, mean, and sometimes last, median, std) preserves the signal needed for the downstream tasks, so real-versus-synthetic gaps can be blamed on the generator rather than on information lost in the summarization. If this premise fails, the paper's time-series conclusions become conclusions about tabular summaries, not about temporal synthesis.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic tables hit 97% real utility, ICU drops to 64%","CoMedBench: 97% tabular utility, 64% ICU AUPRC","Synthetic data: 97% on tables, 64% on ICU events","ICU imbalance crushes synthetic data: 64% vs 97% on tables","Synthetic EHRs: near-real for tables, 64% for ICU events"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001152,"raw_usage":{"total_tokens":4864,"prompt_tokens":1125,"completion_tokens":3739,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":741,"completion_tokens_details":{"reasoning_tokens":3645}},"tokens_in":741,"tokens_out":3739,"duration_ms":24522,"temperature":1.0,"reasoning_tokens":3645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:55:04.099752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the benchmark's TRTR protocol on real ICU stays twice: once on the paper's per-stay summary features and once on the raw multivariate time series with a sequence model such as a GRU-D or transformer. If the sequence model materially beats the summary model on real training data for mortality or readmission, the summary representation itself discards task-relevant temporal signal and the reported time-series utility gaps cannot be attributed to the synthesizers; if the two match, the paper's interpretation survives.","supporting_citations":[],"review_version":1}