{"id":"c632adfa-86d4-470b-bc9f-7de50e9774be","arxiv_id":"1908.11463","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A CNN-LSTM model trained on 8,682 polysomnograms stages sleep from ECG and abdominal respiration with Cohen's kappa of 0.600 (5 stages) and 0.762 (wake/NREM/REM).","lead":"This study shows that deep learning can stage sleep from ECG and breathing signals alone, reaching moderate agreement with expert scoring on a large dataset. The method could enable sleep tracking in settings where brain monitoring is unavailable, such as wearables and ICUs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PSG-level split likely places same patients in training and test; reported kappa may be inflated by patient-specific leakage.","rationale":"The paper's central claim is a numerical performance level, and that number is only meaningful if the test set is an independent sample. The authors split by PSG rather than by patient and do not report overlap, so the reported kappa cannot be interpreted as generalizable performance for unseen patients. The overlap is not hypothetical: with 8,682 PSGs and 7,208 patients, random PSG-level assignment makes substantial train/test patient sharing a near certainty. The reader's weakest assumption identifies exactly this issue, and I agree with it. The concern is addressable: the authors can report the overlap and a patient-stratified reanalysis. If the reanalysis is not performed, the claim 'substantial information about sleep stages in a large population' should be weakened to note that many individuals appear in both training and test, which is materially different. I would not reject the paper: the method is described in detail, the dataset is unusually large, and even under some leakage the absolute kappa likely remains moderate. But the headline numbers need qualification. The reader's CONDITIONAL verdict is therefore appropriate; my read does not change it.","tokens_in":13480,"tokens_out":4315,"duration_ms":44490,"concrete_test":"Obtain patient IDs for all 8,682 PSGs and count how many of the 1,000 test PSGs belong to patients who also have a PSG in the training or validation set. Then re-run the best ECG+ABD pipeline with a patient-stratified split: assign all PSGs of each patient to the same set (for example, 80/10/10 by patient), retrain CNN and LSTM with the same hyperparameters and loss weighting, and recompute the Table 2 kappas. If the five-stage kappa drops materially outside the original bootstrap confidence interval, or specifically below roughly 0.58, the original PSG-level split inflated performance and the claim should be qualified accordingly. Report both the overlap count and the patient-stratified kappas.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that cardiorespiratory signals alone yield kappa 0.600 for five-stage staging and 0.762 for W/NREM/REM. The validity of this number depends on the test set being independent of training. The Methods state: 'We randomly split the PSGs into a training set of 6,682 PSGs, a validation set of 1,000 PSGs and a testing set of 1,000 PSGs,' while the dataset contains 8,682 PSGs from only 7,208 patients. Therefore at least 1,474 PSGs are additional recordings from patients who already contribute another PSG. A random PSG-level split makes same-patient train/test overlap nearly certain: if most repeats are second nights, roughly 20-25% of the 1,000 test PSGs could come from patients whose other recording is in the training set. Such test PSGs are not independent; the CNN and LSTM can exploit patient-specific ECG/respiratory signatures, sleep fragmentation patterns, or technician scoring style, inflating accuracy. The number of overlapping patients is not reported, and no patient-stratified analysis is provided. The authors' own limitation (4) acknowledges only that '1-fold validation' is used and that cross-validation would be 'less biased'; the more consequential patient-overlap issue is not mentioned. Since the abstract's conclusion is framed as validation 'in a large population,' the independence of the test set is load-bearing for the headline performance numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes deep neural network models (CNN plus bidirectional LSTM) for automatic sleep staging from ECG and/or respiratory effort signals. Using a clinical dataset of 8,682 polysomnograms from 7,208 patients, the authors train five networks (ECG, chest effort, abdominal effort, ECG+chest, ECG+abdomen) and report held-out test performance on 1,000 PSGs. The best model, ECG+abdomen, achieves a five-stage Cohen's kappa of 0.600 (95% CI 0.599--0.602) and a three-stage (W/NREM/REM) kappa of 0.762 (0.760--0.763). Subgroup analyses examine age, sex, BMI, AHI, PLM, and medication effects.","tokens_in":13804,"tokens_out":5736,"duration_ms":51342,"significance":"If the reported test-set estimates are unbiased, this is a valuable contribution: it demonstrates on a large, heterogeneous clinical population that cardiorespiratory signals carry substantial sleep-stage information and it provides a practical benchmark for non-EEG sleep staging. The paper includes useful comparisons to prior work and appropriate robustness checks (R-peak jitter, group-specific kappas, t-SNE, and signal examples). However, the validity of the headline kappa values depends critically on the independence of the test set from the training set; the current PSG-level split does not guarantee patient-level independence, so the numerical claims must be re-examined with a patient-disjoint split. The work is therefore significant conditionally on this being addressed.","major_comments":[{"comment":"The random split is performed at the level of PSG ('We randomly split the PSGs into a training set of 6,682 PSGs, a validation set of 1,000 PSGs and a testing set of 1,000 PSGs'), but Table 1 reports 8,682 PSGs from 7,208 patients, so at least 1,474 PSGs are additional recordings from patients who already contribute another PSG. With a PSG-level split, it is highly likely that the same patient appears in both training and test sets, allowing the model to exploit patient-specific ECG/respiratory signatures or technician scoring style and inflate the reported kappa. The manuscript does not report the number of overlapping patients or any patient-stratified analysis. Since the abstract's conclusion is that performance is validated 'in a large population,' the test set must be independent at the level of the patient; please provide a patient-disjoint evaluation (e.g., retrain on a patient-stratified split or evaluate the current model on a subset of test PSGs from patients not seen in training) and report the kappa values and overlap statistics.","section":"Methods, Training and Evaluating the Network"},{"comment":"The bootstrap confidence intervals are described only as 'sampling with replacement 1,000 times' without specifying whether the resampling unit is an epoch, a PSG, or a patient. Given the hierarchical structure of the data (epochs within PSGs, PSGs within patients), epoch-level bootstrapping would severely underestimate the variance of kappa. Please specify the resampling unit and, if it is not the PSG, recompute the intervals by bootstrapping PSGs (or patients) to reflect the actual sampling variability. This is important because the narrow confidence intervals in Table 2 are used to support claims about the precision of the reported kappa values.","section":"Methods, Training and Evaluating the Network"}],"minor_comments":[{"comment":"The statement 'Since the testing set has 1,000 PSGs (6.6×10^5 30-second epochs), the confidence interval is narrow. Therefore the differences between kappa values are all significant at 0.05 level' is not justified by the presented CIs; overlapping CIs do not imply significance, and a paired test (e.g., per-PSG difference in kappa with a bootstrap) is needed for this claim.","section":"Results, Table 2"},{"comment":"Please specify how many PSGs per patient and the distribution of repeated PSGs; this is needed to understand the potential overlap between the training and test sets.","section":"Methods, Dataset"},{"comment":"Please define the x-axis in Figure S3; the caption does not state the unit of the jitter standard deviation used in the robustness test.","section":"Figure S3"},{"comment":"The sentence 'Only one prior study used more than 100 participants for training and evaluation' should be clarified with respect to Table 4, since the Radha et al. entry reports 352 ECG participants; please specify whether the count refers to studies with more than 100 participants that also used deep learning.","section":"Discussion"},{"comment":"The abstract says 'using a dataset including 8,682 polysomnographs' but the Methods section describes exclusion criteria; please clarify that 8,682 is the number of PSGs remaining after exclusions.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The patient-overlap issue is the key obstacle to publication. If the authors can provide a patient-disjoint evaluation (or quantify the overlap and convincingly show no inflation), the paper could become acceptable. The self-citations to prior work from the same group are appropriate and not excessive. The topic fits the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the largest cardiorespiratory sleep-staging study I've seen, and the central finding—that ECG plus abdominal effort carries enough information to stage sleep at kappa around 0.6—is probably real. But the reported numbers are likely somewhat inflated, because the train/test split was done at the PSG level and the dataset has multiple PSGs per patient. That's the main thing to know before citing the headline kappa.\n\nThe paper earns credit. 8,682 PSGs is a clear jump over prior work, and the end-to-end CNN-LSTM on raw R-peak timings and respiratory waveforms is a sensible modern approach. The confusion matrices and subgroup analyses are reported honestly: performance drops with age and AHI, as expected, and the REM accuracy of 92% is genuinely useful. The methods are described in enough detail to be replicated if the data were available, though no code or data are shipped.\n\nThe soft spots are real but addressable. The split is described as \"We randomly split the PSGs,\" and with 8,682 PSGs from 7,208 patients, a patient-level split is not used. The authors don't report how many patients appear on both sides. Given that the LSTM learns temporal patterns from 10-minute sequences, it can easily pick up patient-specific ECG morphology, sleep fragmentation, or even technician scoring style. The bootstrap confidence intervals treat epochs as independent, which further understates uncertainty. In the limitations they mention 1-fold validation but not the patient-overlap issue, which is the more consequential point.\n\nThat said, I don't think the qualitative conclusion is in jeopardy. Even a conservative reading—call it kappa 0.5 to 0.55 on truly independent patients—still supports the claim that cardiorespiratory signals are informative for sleep staging in a large, heterogeneous population. The paper just isn't the final word on external validity; an external validation or patient-stratified analysis would settle it.\n\nI'd send this to peer review, ideally with a request for patient-level analysis or external data. It's a solid engineering contribution with a transparent write-up, and the leak concern is fixable rather than fatal. I'd cite it for the scale and the architecture comparison, but I'd avoid quoting the kappa without a caveat. Reading group? Maybe, if the group cares about medical ML evaluation pitfalls.","headline":"Large-scale cardiorespiratory sleep staging study with a real patient-leakage concern; the headline kappa is likely somewhat optimistic but the qualitative claim that ECG and respiration carry substantial sleep-stage information holds.","tokens_in":14363,"tokens_out":2716,"would_cite":true,"duration_ms":25663,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cardiorespiratory signals alone can stage sleep with moderate accuracy in a large clinical population, reaching a five-stage kappa of 0.600.","keywords":["sleep staging","deep learning","electrocardiography","respiration","polysomnography","convolutional neural network","long short-term memory","Cohen's kappa"],"falsifier":"Re-run the experiment with a strict patient-level split—all recordings from each patient held out of training—and compare five-stage kappa; a meaningful drop below 0.600 would show the current number depends on overlapping patients between sets. Scoring an external cohort of ECG and respiratory recordings from a different institution or from a wearable device would provide the same sort of check.","tokens_in":13309,"feed_emoji":"💤","tokens_out":9735,"duration_ms":85833,"temperature":0.7,"pith_summary":"This paper sets out to prove that sleep can be staged automatically from the heart's electrical activity and breathing patterns, without any EEG. Using 8,682 clinical polysomnograms (overnight sleep recordings), the authors train convolutional and long short-term memory networks and report the best five-stage performance with ECG plus abdominal respiratory effort: a chance-corrected agreement score (Cohen's kappa) of 0.600, and 0.762 when the task is collapsed to awake, NREM, and REM. Because the test set contains about 660,000 epochs, the differences between input-signal combinations are statistically clear-cut. If the result holds, sleep staging becomes possible from signals already recorded in hospitals and many wearable devices, which matters where EEG is unavailable or impractical.","feed_headline":"Heart and breathing alone stage sleep at kappa 0.600","feed_subtitle":"A deep network trained on 8,682 overnight recordings separates wake, NREM, and REM without EEG.","key_machinery":"The engine is a two-part deep network: a residual convolutional network that learns features from a 270-second window centered on each 30-second epoch to be scored, followed by a bidirectional long short-term memory network that models temporal context across consecutive epochs. The ECG is represented as a binary sequence of R-peak times, and respiratory effort signals are downsampled to 10 Hz; artifactual epochs are removed for CNN training but kept for the LSTM so temporal continuity is preserved. The loss is a class-weighted cross-entropy that counteracts the dominance of N2, and the configuration is selected on a validation set. This architecture is what converts heart-rate and breathing dynamics into sleep-stage probabilities.","core_discovery":"The paper claims that cardiorespiratory signals contain enough information for moderately accurate automatic sleep staging in a large, heterogeneous adult population. Combining R-peak timing from the ECG with the abdominal respiratory effort trace, a deep network reproduces the five clinical sleep stages with a Cohen's kappa of 0.600 and separates wake, NREM, and REM with kappa 0.762. The confusion matrix shows REM epochs are recalled at 92.2%, while most mistakes fall between adjacent or transitional stages, such as wake/N1, N1/N2, and N2/N3. Performance is better in younger participants and in those with a low apnea-hypopnea index, while remaining stable across a wide range of common outpatient medications. The authors read the age and apnea dependence as partly biological: EEG-defined N2 and N3 can share cardiorespiratory signatures, so some apparent errors may reflect real differences between autonomic state and EEG scoring.","pith_inferences":["A patient-level split—holding out every recording of a patient from training—would be the natural stress test of whether kappa 0.600 reflects generalization to unseen people rather than to unseen recordings.","The network's medication robustness hints that transfer learning from this dataset could bootstrap sleep staging on ICU or wearable ECG, where full polysomnography is infeasible.","The systematic disagreement between EEG-based and cardiorespiratory-based staging in older adults and severe apnea could itself become a marker of sleep fragmentation or altered autonomic-cardiorespiratory coupling.","The same architecture could be adapted to predict continuous sleep-depth measures, such as cardiopulmonary coupling, rather than only discrete stages."],"forward_implications":["Sleep staging without EEG becomes practical in intensive care, ambulatory monitoring, and consumer wearables, as long as an ECG lead and a respiratory band are available.","ECG plus abdominal effort is the signal pairing to prefer for cardiorespiratory staging; it beats ECG alone, abdominal effort alone, and ECG plus chest effort.","Collapsing sleep into three states (awake, NREM, REM) gives a kappa of 0.762, so coarse sleep tracking is substantially more reliable than five-stage staging and is the more defensible target for wearables.","Five-stage performance remains moderate even in severe sleep apnea (kappa 0.574), so the approach is not broken by common disordered breathing, though it may need extra signals or calibration for older and higher-AHI patients.","With REM recall at 92.2%, cardiorespiratory features separate REM especially cleanly, so REM detection is a strong building block for downstream sleep analysis."],"supporting_citations":[{"why":"Defines the clinical scoring standard for the five sleep stages used as ground truth.","marker":"[1]"},{"why":"Establishes the prior EEG-based deep sleep staging result that frames the present comparison.","marker":"[17]"},{"why":"Supplies the QRS detection method that converts the ECG into R-peak timing.","marker":"[18]"},{"why":"Supplies the artifact filter used to remove spurious R-peaks before training.","marker":"[19]"},{"why":"Provides the convolutional network architecture adapted for cardiorespiratory inputs.","marker":"[20]"},{"why":"A prior ECG-plus-respiratory-effort staging study whose smaller scale motivates the large-dataset approach.","marker":"[23]"},{"why":"A prior ECG/PPG staging result used as a performance baseline in the comparison table.","marker":"[26]"}],"fun_headline_variants":["ECG plus effort stages five sleep stages at kappa 0.600","Wake, REM, NREM separated at kappa 0.762 without EEG","Deep learning from heart and breathing hits kappa 0.600","Sleep staging from cardiorespiratory signals alone, kappa 0.600"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that splitting the 8,682 polysomnograms at the recording level, rather than at the patient level, keeps evaluation independent; if the same patient has several recordings and they appear in both training and testing, the reported agreement could be optimistic for new patients.","fun_headline_variants_meta":{"raw":{"variants":["ECG plus effort stages five sleep stages at kappa 0.600","Wake, REM, NREM separated at kappa 0.762 without EEG","Deep learning from heart and breathing hits kappa 0.600","Sleep staging from cardiorespiratory signals alone, kappa 0.600"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001117,"raw_usage":{"total_tokens":4671,"prompt_tokens":989,"completion_tokens":3682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":3599}},"tokens_in":605,"tokens_out":3682,"duration_ms":27451,"temperature":1.0,"reasoning_tokens":3599,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:13:52.218434+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the experiment with a strict patient-level split—all recordings from each patient held out of training—and compare five-stage kappa; a meaningful drop below 0.600 would show the current number depends on overlapping patients between sets. Scoring an external cohort of ECG and respiratory recordings from a different institution or from a wearable device would provide the same sort of check.","supporting_citations":[{"cited_title":"The visual scoring of sleep in adults","cited_arxiv_id":null,"evidence_quote":"Defines the clinical scoring standard for the five sleep stages used as ground truth."},{"cited_title":"Expert-level sleep scoring with deep neural networks","cited_arxiv_id":null,"evidence_quote":"Establishes the prior EEG-based deep sleep staging result that frames the present comparison."},{"cited_title":"A real-time QRS detection algorithm","cited_arxiv_id":null,"evidence_quote":"Supplies the QRS detection method that converts the ECG into R-peak timing."},{"cited_title":"ADARRI: a novel method to detect spurious R-peaks in the electrocardiogram for heart rate variability analysis in the intensive care unit","cited_arxiv_id":null,"evidence_quote":"Supplies the artifact filter used to remove spurious R-peaks before training."},{"cited_title":"Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network","cited_arxiv_id":null,"evidence_quote":"Provides the convolutional network architecture adapted for cardiorespiratory inputs."},{"cited_title":"Sleep stage classification with ECG and respiratory effort","cited_arxiv_id":null,"evidence_quote":"A prior ECG-plus-respiratory-effort staging study whose smaller scale motivates the large-dataset approach."},{"cited_title":"LSTM knowledge transfer for HRV- based sleep staging","cited_arxiv_id":null,"evidence_quote":"A prior ECG/PPG staging result used as a performance baseline in the comparison table."}],"review_version":1}