{"id":"b8f1c804-a33f-4c29-bcc0-83aea67eee1b","arxiv_id":"2411.18456","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Synthetic ECG data from diffusion and VQ-VAE models provides only marginal classification gains on individual datasets, a small boost when datasets are merged, and cannot replace real data in transfer learning.","lead":"This paper tests whether synthetic ECG signals generated by three deep generative models can improve arrhythmia classification through data augmentation and transfer learning. It finds that synthetic data helps only slightly when two datasets are merged, and that models pretrained on synthetic data still underperform models trained on real data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transfer-learning conclusion rests on a frozen-feature linear probe (§7.1 item 2); unfreezing the feature extractor could erase the reported gap, so the negative claim is not yet established.","rationale":"The reader's weakest_assumption is the load-bearing point. The paper has two headline results: a modest positive augmentation effect on the merged dataset and a negative transfer-learning result. The positive effect is small and reported without error bars or augmentation ratios, which is a real reporting gap, but the negative transfer claim is the stronger and more general conclusion in the abstract and conclusions. That claim is derived exclusively from a linear-probe protocol: freezing every layer except the dense head means the real data can only recalibrate the final classifier, so the comparison measures the linear separability of the synthetic feature space, not the transferability of the learned representation. In standard transfer learning, unfreezing later blocks or all layers with a small learning rate is the default, and it can substantially close gaps that exist under linear probing. A single controlled rerun with unfrozen layers would either confirm the protocol-independent conclusion or show that it is an artifact. This supports keeping the CONDITIONAL verdict; no code is supplied, but the proposed experiment is within the paper's own compute budget (the existing grid is 25 seeds × 5 proportions × 2 or 3 datasets) and would settle the issue.","tokens_in":20684,"tokens_out":9227,"duration_ms":88318,"concrete_test":"On PTB-XL and PTB-XL+CHAPMAN, rerun the 20/40/60/80/100% fine-tuning grid with the same seeds and real-data proportions, but unfreeze all layers (or at least the last two residual blocks) using a small learning rate (e.g., 1e-4 with cosine decay and the same early-stopping/validation protocol) after a short head-only warm-up. Compare final metrics to the real-only baseline with 95% CIs over the 25 seeds. If the Time-VQVAE-pretrained model at 100% real data closes the gap to within the confidence interval—or exceeds the baseline—the paper's transfer-learning conclusion is an artifact of frozen-feature adaptation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative claim—synthetic pretraining is 'not powerful enough to achieve results close to a classifier trained with real data only'—is supported only by the fine-tuning protocol in §7.1 item 2, which freezes all layers except the dense classifier head. That is a linear-probe evaluation, not full transfer learning. If the convolutional feature extractor is allowed to adapt to real data, a synthetic-pretrained model can recover features that are poor for linear probing but useful after adaptation. Tables 6, 8, and 10 therefore demonstrate that head-only fine-tuning underperforms training from scratch; they do not demonstrate that synthetic pretraining cannot replace real training. The abstract and conclusions generalize the protocol-specific result without testing the standard alternative, so the most important negative conclusion is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether synthetic ECG signals generated by three deep generative models (Diffweave, Time-Diffusion, and Time-VQVAE) can be used for data augmentation and transfer learning in arrhythmia classification. The authors train the generative models on PTB-XL, CHAPMAN, and a merged dataset, and evaluate the synthetic data by training a 1D CNN classifier under different train/test combinations (real, synthetic, and hybrid settings), as well as by fine-tuning a synthetically pretrained classifier with increasing proportions of real data. They report that augmentation with synthetic data gives barely noticeable improvements on individual datasets, but that on the merged dataset using synthetic samples as augmented data increases all metrics; they also report that synthetic pretraining followed by head-only fine-tuning does not reach the performance of real-data-only training, with Time-VQVAE performing best among the generative models.","tokens_in":20761,"tokens_out":4918,"duration_ms":40144,"significance":"If the results were fully supported, the paper would provide a useful empirical benchmark on the value of state-of-the-art generative models for ECG augmentation and a cautionary negative result on synthetic pretraining. The study is valuable in scope: it uses two widely used public datasets, compares three open-source generative models, and runs the classification experiments 25 times with multiple metrics. However, the absence of variance reporting, the unspecified synthetic-to-real mixing ratio, and the linear-probe-only transfer protocol mean that the two headline claims are not yet quantitatively established. The work could become a solid reference for the community after these issues are addressed, but in its current form the evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The central positive claim that merging PTB-XL and CHAPMAN with synthetic data increases all metrics under the TrRSTeR setting is not quantitatively supported: the manuscript never reports how many synthetic samples were added relative to the real training set, so the improvement from 0.8574 to 0.8673 accuracy could be due to dataset size or class-balance changes rather than to the quality of the synthetic data. In addition, despite stating in Section 8 that all experiments were repeated n=25 times, Tables 5-10 report only point estimates with no standard deviations or confidence intervals, making it impossible to judge whether the observed differences are statistically significant. The authors should report the mixing ratio and the variance of the 25 repeats.","section":"Section 9.3, Table 9"},{"comment":"The transfer-learning conclusion is based on a protocol that freezes all layers except the dense classifier head during fine-tuning. This is a linear-probe evaluation: it tests how linearly separable the frozen synthetic-pretrained features are, not whether the pretrained model can be adapted to real data. The paper does not run the standard full fine-tuning alternative in which the feature extractor is also updated, so the negative claim that synthetic pretraining is 'not powerful enough to achieve results close to a classifier trained with real data only' is underdetermined. The authors should either report full fine-tuning results or substantially weaken the conclusion in the abstract and Section 10.1.","section":"Section 7.1, item 2; Tables 6, 8, 10"},{"comment":"The experimental setup omits details necessary for reproducibility and for assessing potential leakage or bias: there is no description of how the 25 repeats were constructed, how the train/test splits were stratified (e.g., per patient), whether the same test set was reused across repeats and across generative models, or whether the 500 Hz data were resampled to 100 Hz and how the signals were preprocessed (filtering, normalization, segmentation). These details are essential because the comparison between generative models and between real and synthetic settings depends on identical evaluation conditions.","section":"Section 8"},{"comment":"The paper introduces several quality metrics (2-sample test classification score, MMD, and dimensionality-reduction visualizations) but never reports their quantitative results; Section 10.3 then draws conclusions such as 'MMD and visualization techniques lack the power to differentiate synthetic data from real one' without showing the corresponding data. If these metrics are part of the evaluation, their results should be presented; otherwise the claims in Section 10.3 should be removed or explicitly marked as qualitative impressions.","section":"Section 7, Section 10.3"}],"minor_comments":[{"comment":"The sentence listing the diffusion models is a fragment: 'We opted for three different architectures, namely Diffwave, [54] Diffusion-TS [55] and the Unet1D conditional model from Huggingface.' Also, the name 'Diffweave' in the abstract and Table 5 is inconsistent with 'Diffwave' used in the text.","section":"Section 5.1"},{"comment":"There is a typo in 'Excution Time' (should be 'Execution Time') in Table 6 and in the surrounding discussion.","section":"Section 9.1"},{"comment":"The sentence 'because PTB-XL and CHAPMAN have different number of samples in each class (4 and it is expected to have an effect...' is garbled and should be rewritten.","section":"Section 9.3"},{"comment":"The description of the 2-sample test classification score is ambiguous: it says 'A classifier trained with real data is used in order to discriminate between real data and synthetic data,' but a classifier trained only on real data cannot be used to discriminate real from synthetic without also training on synthetic labels; please clarify the intended procedure.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The paper's two headline claims rest on experimental details that are not reported. The merged-dataset augmentation improvement is reported without a mixing ratio or variance, and the transfer-learning negative result is only tested under a frozen-feature protocol. Both are fixable within the scope of the manuscript, but the authors must add the missing descriptions and ideally run the full fine-tuning condition. The paper is within the scope of cs.LG and is likely to be of interest to the ECG generation community, but in its current form the evidence is not complete."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a systematic empirical comparison of three known generative models (SSSD-ECG, Unet1D diffusion, Time-VQVAE) for ECG augmentation and transfer learning on PTB-XL, Chapman, and their merge. The broad negative result is likely robust: synthetic data alone does not transfer well to real data, and augmentation gives barely noticeable gains on individual datasets. On the merged dataset, adding pooled synthetic samples does improve all metrics slightly, but the effect is small and the abstract overstates it.\n\nThe evaluation protocol is a real strength. Multiple train/test settings, fine-tuning with increasing real data proportions, and a consistent classifier make this a legitimate map of where synthetic ECG data does and does not help. The observation that MMD and visualizations fail to tell real from synthetic while classifier-based two-sample tests succeed is a useful side result. The authors are also transparent that their fine-tuning freezes all but the classifier head.\n\nNow the soft spots. The central augmentation claim lacks quantitative support: the synthetic-to-real mixing ratio is never reported, no variance is given despite 25 repeats, and preprocessing and split details are absent. The abstract's \"increase across all metrics\" holds for the pooled synthetic samples but not for every individual model—Time-VQVAE precision drops on the merged set. The transfer-learning conclusion is underdetermined: freezing the feature extractor makes it a linear probe, so the gap between synthetic-pretrained and real-trained models could shrink if the backbone were fine-tuned. That concern is real and not addressed. Minor issues: no code or configs, and the Diffweave/diffwave naming is confusing.\n\nWho is this for? Practitioners deciding whether synthetic ECG augmentation is worth the effort. The answer here is \"only slightly, and only on merged data,\" which is useful even if not headline-grabbing. The paper deserves a serious referee because the empirical comparison is valuable and the claims are fixable. I would send it to review, but require the missing details (mixing ratio, variance, preprocessing, splits) and a discussion of the unfrozen fine-tuning alternative—or ideally a quick experiment with the backbone unfrozen.","headline":"A useful but under-specified empirical comparison of three ECG generative models; the augmentation gains are marginal and the transfer-learning negative rests on a frozen-feature protocol.","tokens_in":21351,"tokens_out":2766,"would_cite":false,"duration_ms":25239,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper tests whether synthetic ECG data generated by three deep generative models can augment real arrhythmia datasets and support transfer learning, and finds that gains appear only when the real datasets are merged.","keywords":["synthetic ECG","data augmentation","transfer learning","arrhythmia classification","diffusion models","Time-VQVAE","PTB-XL","Chapman dataset"],"falsifier":"Unfreeze all layers during fine-tuning on real data and rerun the transfer experiments; if a synthetically pretrained model then reaches or exceeds the real-only classifier once it sees enough real data, the paper's negative transfer-learning conclusion is overturned.","tokens_in":20418,"feed_emoji":"❤️","tokens_out":5829,"duration_ms":47837,"temperature":0.7,"pith_summary":"This paper asks whether synthetic electrocardiogram (ECG) traces generated by deep learning models can stand in for scarce, privacy-protected real recordings. The authors train three generative models—Diffweave, Time-Diffusion, and Time-VQVAE—on the PTB-XL and Chapman datasets, then use the synthetic samples to augment arrhythmia classification and to pretrain a classifier that is fine-tuned with increasing amounts of real data. The central finding is conditional: augmenting either dataset alone with synthetic samples gives barely noticeable improvements, but when both datasets are merged, every evaluated metric improves across classifiers. For transfer learning, synthetic pretraining never matches a classifier trained only on real data, with Time-VQVAE the least bad option. The paper also shows that common distribution-closeness measures such as MMD and dimensionality-reduction visualizations cannot separate synthetic from real ECGs, while classifier-based discrimination can.","feed_headline":"Synthetic ECGs sharpen merged-dataset classifiers, not single ones","feed_subtitle":"Best of three generators, Time-VQVAE, still trails real-only training; augmentation pays only after merging PTB-XL and Chapman.","key_machinery":"The machinery is a three-way comparison of generative architectures plus a split-train/evaluate protocol. Diffweave is a non-autoregressive diffusion model originally built for audio; Time-Diffusion is a U-Net-shaped diffusion model with a label-conditioning scheme; Time-VQVAE is a vector-quantized variational autoencoder that learns discrete low- and high-frequency latent codes in the STFT domain and samples them with a bidirectional transformer prior. Each model is trained per dataset and on the merged dataset. The classification evaluation uses a 1D residual CNN with hyperparameters tuned per dataset, and applies five train/test splits—real-only, synthetic-only, train-synthetic/evaluate-real, train-real/evaluate-synthetic, and real-plus-synthetic train/evaluate-real—plus a fine-tuning protocol that freezes feature layers and retrains only the dense classifier on real data in 20% increments. The splits are what make the boundary visible: synthetic-only classifiers look good on synthetic tests but collapse on real tests, while augmentation gains appear only in the merged-data real-plus-synthetic setting.","core_discovery":"On the paper's own terms, the discovery is that current state-of-the-art generative models produce ECG samples that visually and statistically resemble real ones, yet the samples are only useful for data augmentation when the real training pool already spans diverse sources. In the train-on-real-plus-synthetic/evaluate-on-real setting, merging PTB-XL and Chapman before augmentation raises accuracy, precision, recall, f1, and ROC AUC across all three generators compared with real-only training; the same augmentation on either dataset individually leaves scores essentially unchanged. In transfer learning, a model pretrained on synthetic data and then fine-tuned on increasing fractions of real data—with all but the classifier layers frozen—consistently underperforms a model trained on real data alone, and Time-VQVAE's samples give the best fine-tuned scores of the three. The paper thus establishes a precise boundary: synthetic ECG data helps modestly as an additive to a merged real corpus, but cannot substitute for real data or bootstrap a classifier to real-data-level performance under this protocol.","pith_inferences":["The paper's frozen-feature fine-tuning protocol may understate transferability; a natural extension is to allow the convolutional feature extractor to adapt, which would test whether the negative result is a property of the data or of the protocol.","The fact that augmentation helps only on the merged dataset suggests sample diversity across recording sources matters more than raw sample count; this could be tested by holding the number of real training samples fixed while varying the number of datasets.","The classifier-based real-vs-synthetic discrimination that works in the quality evaluation could double as an authorship detector for generative physiological signals, a direction the paper mentions only as a closing remark.","The same comparison could be run on EEG or other periodic physiological signals to determine whether the gap between synthetic and real data is specific to ECG or general to time-series generation."],"forward_implications":["When both PTB-XL and Chapman are merged, adding synthetic samples from any of the three generators improves every reported classification metric under the real-plus-synthetic training setting.","Time-VQVAE is the generator of choice for transfer: it yields the highest fine-tuned scores across datasets and consistently reduces false negatives when synthetic-trained models are evaluated on real data.","None of the tested generators produces data that can replace real recordings: train-on-synthetic/evaluate-on-real scores remain far below real-only baselines.","Classifier-based discrimination is a stricter and more informative quality check than MMD or dimensionality-reduction visualization, which fail to separate synthetic from real ECGs.","Fine-tuning a synthetically pretrained model converges in about half the time of training from scratch on real data, so the practical payoff of synthetic pretraining is computational speed, not accuracy."],"supporting_citations":[{"why":"Supplies Time-VQVAE, the generative architecture that outperforms the other two in fine-tuning experiments.","marker":"[57]"},{"why":"Supplies Diffweave, one of the three diffusion-based generators compared.","marker":"[54]"},{"why":"Supplies Diffusion-TS, the interpretable diffusion time-series generator used as Time-Diffusion.","marker":"[55]"},{"why":"Supplies the SSSD-ECG model used in place of the original Diffweave audio model for ECG generation.","marker":"[35]"},{"why":"Supplies PTB-XL, one of the two ECG datasets whose real and synthetic samples drive the classification experiments.","marker":"[29]"},{"why":"Supplies the Chapman dataset, the second ECG corpus whose merge with PTB-XL yields the augmentation gains.","marker":"[53]"},{"why":"Supplies the benchmark framing and classifier families used to justify the 1D residual CNN architecture for ECG classification.","marker":"[48]"},{"why":"Supplies the VQ-VAE discrete-latent first stage that Time-VQVAE's two-stage generation builds on.","marker":"[58]"}],"fun_headline_variants":["Synthetic ECGs boost merged datasets, not single ones","Time-VQVAE best synthetic ECG, still trails real data","Generative ECG data: only helps after dataset merging","ECG deepfakes: fine-tuning can't beat real data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that synthetic pretraining cannot match real-data training depends on the fine-tuning protocol that freezes all feature-extraction layers and retrains only the dense classifier; if those layers were allowed to adapt to real data, the gap might narrow or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic ECGs boost merged datasets, not single ones","Time-VQVAE best synthetic ECG, still trails real data","Generative ECG data: only helps after dataset merging","ECG deepfakes: fine-tuning can't beat real data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1276,"prompt_tokens":1042,"completion_tokens":234,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":164}},"tokens_in":658,"tokens_out":234,"duration_ms":2806,"temperature":1.0,"reasoning_tokens":164,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:11:43.133416+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Unfreeze all layers during fine-tuning on real data and rerun the transfer experiments; if a synthetically pretrained model then reaches or exceeds the real-only classifier once it sees enough real data, the paper's negative transfer-learning conclusion is overturned.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Diffweave, one of the three diffusion-based generators compared."},{"cited_title":"Diffusion-based conditional ECG generation with structured state space models,","cited_arxiv_id":null,"evidence_quote":"Supplies the SSSD-ECG model used in place of the original Diffweave audio model for ECG generation."},{"cited_title":"PTB-XL, a large publicly available electrocardiography dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies PTB-XL, one of the two ECG datasets whose real and synthetic samples drive the classification experiments."},{"cited_title":"A 12-lead electrocardiogram database for arrhythmia research covering more than 10,000 patients,","cited_arxiv_id":null,"evidence_quote":"Supplies the Chapman dataset, the second ECG corpus whose merge with PTB-XL yields the augmentation gains."},{"cited_title":"Neural discrete representation learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the VQ-VAE discrete-latent first stage that Time-VQVAE's two-stage generation builds on."}],"review_version":1}