{"id":"f41adf0d-6856-44be-8e47-44e69a04d1dd","arxiv_id":"2508.00131","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Latent codes from a stochastic autoencoder plus ECG summary features predict reduced LVEF with AUROC 0.901, close to a CNN's 0.909, while handling small training sets better.","lead":"ECG recordings are compressed into 30-number latent codes by several autoencoder variants, then a tree-based model predicts cardiac conditions from the codes. The authors report that one variant, the Stochastic Autoencoder, nearly matches a state-of-the-art CNN for detecting low ejection fraction while needing far less labeled data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Potential train/test leakage in autoencoder and scaling could inflate the 0.901 AUROC, but the split ambiguity is checkable.","rationale":"The reader's weakest_assumption is exactly the concern I identify: the model training and normalization appear to use the full dataset, making test encodings not fully out-of-sample. The paper's own text is internally inconsistent on this point (90% split in II-A vs ~1.1 million ECGs in II-C), which is a flag, not proof of leakage. Because the concern is specific, checkable, and central to both the headline AUROC and the small-data robustness claim, the right response is a conditional verdict: the paper should be revised with an explicit statement of which split trained the autoencoder and scaler, and ideally a from-scratch retraining check. I do not move to outright REJECT because the inconsistency is ambiguous and the authors' stated protocol, if implemented as claimed, would support the result. I agree with the reader's identification of the weakest assumption, though I would frame it as one testable ambiguity rather than a confirmed flaw. Other issues (no confidence intervals, inconsistent table sizes, missing code) are secondary relative to this leakage question.","tokens_in":10873,"tokens_out":2014,"duration_ms":15135,"concrete_test":"Check the training log/config or code for the SAE and the global absolute-max scaler: confirm that the scaler constant and the SAE weights were computed using only the 90% train split (unique patient IDs) and never updated with test-set inputs. Equivalently, retrain the SAE from scratch on the 90% split only, recompute the scaler on that split, regenerate all encodings, and re-run the LGBM LVEF experiment; if the combined-feature AUROC drops materially below 0.901 (or below the CNN's 0.909), the headline claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that SAE encodings plus ECG summary features reach AUROC 0.901 on a holdout test set, nearly matching a CNN. This requires the encodings to be genuinely out-of-sample. The paper is ambiguous: Section II-A says ECGs were split by unique patient IDs into training (90%) and test (10%), but Section II-C says models were trained on ~1.1 million ECGs, the size of the full 1,065,368-ECG dataset. If the autoencoder/SAE and the global absolute-max scaler were fit on all data including test patients, then test encodings are not truly out-of-sample and the 0.901 AUROC is inflated, possibly substantially. The same ambiguity affects the signal reconstruction tables and the CNN baseline. This is the single most load-bearing concern because the headline number and the small-data claim (Tables 5-6) both depend on a clean split. It is a correctness risk, not a novelty or style issue. The paper asserts the split but does not state that the autoencoder was trained only on the training split; no code or data release allows verification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three new variational autoencoder variants (Stochastic Autoencoder SAE, Annealed beta-VAE, and Cyclical beta-VAE) for learning compact 30-dimensional ECG latent representations from a large clinical dataset (1,065,368 ECGs). The authors evaluate reconstruction fidelity (MAE, MSE, DTW) and downstream predictive performance using LGBM classifiers trained on the encodings for QRS measurements, bundle branch blocks, and reduced left ventricular ejection fraction (LVEF). The headline claim is that combining SAE encodings with traditional ECG summary features achieves a holdout test AUROC of 0.901 for reduced LVEF, nearly matching a state-of-the-art CNN (0.909) while requiring less computation and degrading less steeply with small training sets.","tokens_in":10995,"tokens_out":4564,"duration_ms":43251,"significance":"If the central claims hold, the paper offers a practically valuable contribution: a compact, interpretable intermediate representation that lets simple tree-based models approach CNN-level performance on ECG classification with far less data and compute. The study uses a very large clinical ECG dataset, compares multiple autoencoder variants, and evaluates on several clinically relevant tasks, which are strengths. The reconstruction quality of the best model (MAE 15.7 microV) is plausibly at signal-noise level. However, the credibility of the headline numbers depends entirely on a clean patient-level train/test split and consistent evaluation protocols. The manuscript currently contains ambiguities about whether the autoencoder and global scaler saw test data, and the reported test-set sizes for the reduced-LVEF task are internally inconsistent. Without resolving these, the 0.901 AUROC and the small-data robustness claims are not verifiable. The lack of confidence intervals and the post-hoc selection of the 'best' model further make the reported differences hard to interpret.","major_comments":[{"comment":"The most load-bearing issue is the potential train/test leakage in the unsupervised feature extractor and the scaling step. Section II-A states that ECGs were split by unique patient IDs into training (90%) and test (10%), but Section II-C says 'Models were trained on ~1.1 million ECGs', which is the size of the entire 1,065,368-ECG dataset rather than the 90% training split. If the autoencoder, PCA, or the global absolute-max scaling constant (II-A: 'dividing by the dataset's maximum absolute amplitude') were fit on the full dataset including test patients, then the test-set encodings are not out-of-sample and the downstream AUROC values in Tables 5 and 6, as well as the reconstruction metrics in Table 1, would be inflated. The manuscript must state explicitly that all unsupervised models and the scaler were fitted only on the training split, and ideally provide the exact training-set size for each model.","section":"III-D, Tables 5 and 6"},{"comment":"The test-set sizes for the reduced-LVEF prediction task are inconsistent. Table 5 reports a test set of n=30,554, while Table 6 reports a holdout test set of n=15,987 for the same task. The text also mentions 303,265 ECGs paired with echocardiography in total for training and testing combined. These numbers do not reconcile, and the discrepancy is not explained. Because the headline comparison (SAE+LGBM AUROC 0.901 vs. CNN AUROC 0.909) appears in Table 6 but the SAE-only results are in Table 5, the reader cannot tell whether the two tables refer to the same test set. This must be clarified, and all models compared on the same holdout set, with exact sample sizes stated.","section":"III-D"},{"comment":"Table 6 does not specify the input representation for each LGBM row. The text claims that 'Combining SAE encodings with traditional ECG features improved the AUROC to 0.901', but the table simply labels the model as 'LGBM' with no indication of whether the input is SAE encodings only, ECG statistics only, or the combination. Since this is the central quantitative claim of the paper, the table and the accompanying text must be aligned so that the exact feature set used for each AUROC value is unambiguous.","section":"III-D"},{"comment":"No confidence intervals or statistical significance tests are provided for any of the AUROC, MAE, or R-squared values. Many of the differences that motivate the narrative are small (e.g., SAE AUROC 0.820 vs. VAE 0.819 in Table 5; RBBB AUROC differences of less than 0.002 in Table 4). Without confidence intervals or error bars, it is impossible to assess whether the reported ordering of methods is meaningful or due to noise. At minimum, the headline AUROC values (0.901 and 0.909) and the reconstruction MAE values in Table 1 need uncertainty quantification, ideally via bootstrap or repeated split resampling.","section":"III"},{"comment":"The paper selects the 'best' autoencoder variant separately for each downstream task and then builds the headline narrative around SAE without correcting for multiple comparisons or pre-registering the model choice. This is a mild selection effect, but combined with the leakage ambiguity it weakens the claim that SAE is the preferred method for reduced-LVEF prediction. The authors should either report results for all models on the same combined feature set or explicitly acknowledge the post-hoc selection and adjust their conclusions accordingly.","section":"II-E"}],"minor_comments":[{"comment":"The running header contains the typo 'Compairson' and should be corrected to 'Comparison'.","section":"Abstract/II-A"},{"comment":"The data description mentions both 500 Hz and 1000 Hz sampling rates: the abstract says '12 leads at 500 Hz', while Section II-A says 'reducing the 1000 Hz 120,000-datapoint 10-sec 12-lead ECG'. Please clarify the native sampling rate and whether the data were resampled to 1000 Hz for the representative-beat extraction.","section":"II-C"},{"comment":"Equation (1) defines z = mu + sigma * epsilon, but the text calls mu and sigma the 'population mean' and 'standard deviation'; these are the encoder outputs for a given input, not population parameters. The notation should be aligned with the standard reparameterization trick notation (e.g., mu_theta(x), sigma_theta(x)).","section":"II-E"},{"comment":"For the SAE, the text says it has a 'stochastic distribution' and beta=0, so no KL term; it is unclear how sigma is trained or whether it collapses. Please clarify the SAE's training objective and whether sigma is a learned parameter or a constant (e.g., a fixed noise level).","section":"II-F"},{"comment":"The CNN baseline is described as 'our implementation of the CNN model architecture from Mayo Clinic' [11], but no architectural details, training hyperparameters, or preprocessing are provided. To make the comparison reproducible, please provide a description or reference to the exact implementation, including how the full 10-sec 12-lead ECG is fed into the network.","section":"III-A"},{"comment":"The paper reports reconstruction metrics in Table 1 for N=1,065,368 (the entire dataset) and in Table 2 for N=97,464, presumably a test subset. It should be stated explicitly whether Table 1 reports training-set reconstructions or test-set reconstructions; if Table 1 includes training data, the claim that the MAE is 'at the level of signal noise' is not an out-of-sample evaluation.","section":"III-A"},{"comment":"The conclusion states that '60,000 data points in a full 10-sec 12-lead ECG at 500 Hz can be reduced to 30 latent variables', which is a useful summary, but the paper should also acknowledge the loss of beat-to-beat information more prominently, as it does in the last paragraph of Section IV.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is written in a conference-paper style with several typos and informal statements, but the underlying dataset is unusually large and the question of whether autoencoder encodings can support simple downstream models is of interest. The main risk is the ambiguity around the train/test split and test-set sizes; if the split is clean, the claims are worth publishing after adding confidence intervals and clarifying the evaluation protocol. I would encourage the editor to ask the authors for a detailed data-flow description and, if possible, a code release to verify the split."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chris — quick note on the Harvey et al. ECG autoencoder paper.\n\nThe headline result is the kind of thing that could actually matter: a 30-number latent code, with an autoencoder trained once on big unlabeled ECGs and a LGBM on the codes, gets AUROC 0.901 for reduced LVEF, close to a Mayo-style CNN at 0.909, and degrades far less when downstream labels are scarce. If that holds up, it's a practical route for institutions without GPU farms. The paper also does something useful that's easy to underrate: it compares seven dimensionality reduction methods (PCA, AE, SAE, VAE, beta-VAE, two schedules) on a million-plus ECG corpus under one preprocessing pipeline, with reconstruction and multiple downstream tasks. The three 'novel' variants are modest — setting beta to 0, cycling beta 0-5, reversing the annealing — but honestly applied and clearly described.\n\nThe soft spot is the one you caught, and it's load-bearing. Section II-A says the dataset was split by patient ID into 90% training and 10% test. Section II-C says the models were trained on ~1.1 million ECGs, which is the full dataset size. If the autoencoder and the global absolute-max scaler saw test-patient ECGs before the encodings were extracted, the test AUROCs aren't out-of-sample and the small-data claim is inflated. The text doesn't explicitly say the unsupervised training was restricted to the training split, and no code/data allow external verification. This is fixable by a clear statement and ideally a re-run with a strictly disjoint autoencoder training set.\n\nTwo smaller issues: the test-set sizes differ between Table 5 (n=30,554) and Table 6 (n=15,987) for the same 'holdout test' without explanation, and there are no confidence intervals anywhere, which matters for an AUROC gap of 0.009. Both are minor relative to the leakage question, but they add up.\n\nMy read: this is a solid applied idea with sloppy reporting on the single most important protocol detail. The reader's reject is defensible as a decision, but not because the method is doomed — it's because the paper doesn't yet prove the result. A serious referee should see this with a request to clarify the split, reconcile the test sizes, add CIs, and release code. I'd like to see the revised version. Bring it to reading group if you want a lively discussion about leakage in self-supervised representations.","headline":"The LVEF result is worth chasing, but the paper as written has an unaddressed train/test ambiguity that keeps it short of publishable.","tokens_in":11648,"tokens_out":4189,"would_cite":false,"duration_ms":36366,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 30-number autoencoder code combined with ECG summary statistics predicts reduced left ventricular ejection fraction with AUROC 0.901, nearly matching a full-signal CNN (0.909) at a fraction of the compute.","keywords":["ECG","variational autoencoder","stochastic autoencoder","LVEF prediction","dimensionality reduction","gradient boosting","latent representation","heart failure screening"],"falsifier":"Retrain the autoencoder and recompute the global absolute-max scaling constant using only the 90% training split, then re-run the reduced-LVEF evaluation; if the AUROC drops materially below 0.901, the original result depended on leakage from test-set ECGs.","tokens_in":10562,"feed_emoji":"❤️","tokens_out":7441,"duration_ms":63727,"temperature":0.7,"pith_summary":"This paper claims that a 30-number encoding of a single representative heartbeat, produced by a stochastic autoencoder and fed to a gradient-boosted tree model together with standard ECG summary statistics, predicts reduced left ventricular ejection fraction (LVEF ≤35%) with an AUROC of 0.901 on a held-out test set. That nearly matches the 0.909 AUROC of a full-signal convolutional network while using far less computation. The same pipeline retains an AUROC of 0.870 with only 10% of the training labels, where the CNN falls to 0.630. If true, this offers a lightweight, data-efficient route to ECG-based heart-failure screening, particularly for small or under-represented datasets.","feed_headline":"30 ECG numbers rival a deep CNN at low-LVEF screening","feed_subtitle":"Compact beat codes plus summary stats reach AUROC 0.901 for heart-failure screening, with far less data than a CNN.","key_machinery":"The machinery is the variational-autoencoder family with a modified ELBO loss: a weighted reconstruction term splits the 750 ms representative beat into P-wave, QRS, and T-wave segments with per-segment weights (θ_P=20, θ_QRS=10, θ_T=15) so the high-amplitude QRS does not dominate, and a KL term with coefficient β controls latent-space regularization. Three variants alter the β schedule: the SAE sets β=0 (pure stochastic reconstruction), the cyclical β-VAE cycles β from 0 to 5 over 20 epochs, and the annealed β-VAE starts at β=10 and anneals to 0. The encoder maps the 2,250-point 3-lead beat to a 30-dimensional latent vector that is decoded back, and that 30-vector is then used as feature input for LGBM downstream classifiers. The load-bearing identity is that 30 latent variables plus summary statistics can carry enough morphological information to rival a full-signal CNN.","core_discovery":"The central discovery is that the stochastic autoencoder (SAE), which drops the KL-regularization term and learns a latent distribution purely to minimize reconstruction loss, produces the most clinically useful 30-variable ECG encodings. When these encodings are concatenated with traditional ECG summary features and fed to an LGBM classifier, the model reaches an AUROC of 0.901 for detecting reduced LVEF, close to the 0.909 of a state-of-the-art CNN trained on raw 10-second 12-lead signals. The paper also finds that a reverse-annealed beta-VAE yields the best signal reconstruction, with a mean absolute error of 15.7 ± 3.2 microvolts, within the level of signal noise. The claim is that these compact encodings preserve morphological details that summary statistics miss, and that the combination can substitute for deep learning on raw signals.","pith_inferences":["The 0.901 figure combines latent features with summary statistics; an ablation separating latent-only from stats-only would reveal how much the latent code contributes.","The representative-beat design discards beat-to-beat rhythm information; encoding the full 10-second trace, flagged by the paper as future work, would be the direct test of whether the approach extends to arrhythmias.","Because the SAE omits KL regularization and still wins, a smooth latent space appears unnecessary for these discriminative tasks; this could be tested on other ECG endpoints.","A clinic-facing extension would be to replace LGBM with a simpler linear model on the same encodings to see how much of the gain depends on tree nonlinearity."],"forward_implications":["A 30-number ECG code plus standard summary statistics can screen for reduced ejection fraction at nearly CNN-level accuracy (0.901 vs 0.909) with far less computation.","The pipeline keeps an AUROC of 0.870 with only 10% of the training labels, where the CNN falls to 0.630, making it suited to rare conditions and small datasets.","Dropping the KL term (SAE) helps downstream prediction, while reverse annealing from β=10 to 0 gives the best reconstruction (MAE 15.7 µV), showing reconstruction and prediction are separable goals.","The same encodings predict QRS duration, amplitudes, and bundle-branch blocks (e.g., SAE LBBB AUROC 0.9701), indicating the 30 numbers retain general morphological information."],"supporting_citations":[{"why":"Supplies the linear transform from the eight independent ECG leads to three orthogonal leads used to compress the signal before encoding.","marker":"[4]"},{"why":"Supplies the reparameterization trick and variational autoencoder formulation used for latent sampling.","marker":"[7]"},{"why":"Defines the KL divergence used for latent-space regularization in the loss.","marker":"[8]"},{"why":"Provides the β-VAE formulation that motivates the β term in the loss.","marker":"[9]"},{"why":"Introduces cyclical annealing of β, which the paper modifies to a 0-to-5 cycle.","marker":"[10]"},{"why":"Is the state-of-the-art CNN architecture on raw ECGs that serves as the performance baseline.","marker":"[11]"},{"why":"Provides external context for the CNN's published AUROC range.","marker":"[13]"},{"why":"Supplies the gradient-boosting classifier used for all downstream tasks.","marker":"[14]"},{"why":"Provides the standardized criteria used to define RBBB and LBBB labels.","marker":"[15]"}],"fun_headline_variants":["30 ECG latent features nearly match CNN for low-LVEF","Compact autoencoder ECG codes rival deep CNN with less data","30-variable ECG encodings hit 0.901 AUROC for reduced LVEF","Stochastic autoencoder yields compact ECG features for screening","ECG autoencoder codes approach CNN accuracy with fewer resources"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported AUROCs are only valid if the autoencoder and the global scaling constant are fit exclusively on the 90% training split, so that test-set encodings are truly out-of-sample.","fun_headline_variants_meta":{"raw":{"variants":["30 ECG latent features nearly match CNN for low-LVEF","Compact autoencoder ECG codes rival deep CNN with less data","30-variable ECG encodings hit 0.901 AUROC for reduced LVEF","Stochastic autoencoder yields compact ECG features for screening","ECG autoencoder codes approach CNN accuracy with fewer resources"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000976,"raw_usage":{"total_tokens":4207,"prompt_tokens":1069,"completion_tokens":3138,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":3050}},"tokens_in":685,"tokens_out":3138,"duration_ms":21346,"temperature":1.0,"reasoning_tokens":3050,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:21:32.418997+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the autoencoder and recompute the global absolute-max scaling constant using only the 90% training split, then re-run the reduced-LVEF evaluation; if the AUROC drops materially below 0.901, the original result depended on leakage from test-set ECGs.","supporting_citations":[{"cited_title":"beta- V AE: Learning basic visual concepts with a constrained variational framework,","cited_arxiv_id":null,"evidence_quote":"Provides the β-VAE formulation that motivates the β term in the loss."},{"cited_title":"Reconstruction of the frank vectorcardiogram from standard electrocardiographic leads: diagnostic comparison of different methods,","cited_arxiv_id":null,"evidence_quote":"Supplies the linear transform from the eight independent ECG leads to three orthogonal leads used to compress the signal before encoding."},{"cited_title":"Screening for cardiac contractile dysfunction using an artificial intelligence–enabled electrocardiogram,","cited_arxiv_id":null,"evidence_quote":"Is the state-of-the-art CNN architecture on raw ECGs that serves as the performance baseline."},{"cited_title":"Artificial intelligence algorithm for screening heart failure with reduced ejection fraction using electrocardiography,","cited_arxiv_id":null,"evidence_quote":"Provides external context for the CNN's published AUROC range."},{"cited_title":"Lightgbm: A highly efficient gradient boosting decision tree,","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-boosting classifier used for all downstream tasks."},{"cited_title":"Surawicz, R","cited_arxiv_id":null,"evidence_quote":"Provides the standardized criteria used to define RBBB and LBBB labels."}],"review_version":1}