{"id":"d1d03c52-5f65-4e2f-bad1-592b46ce4d95","arxiv_id":"2603.00190","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Channel-masked self-supervised pretraining on a 166,500-hour multi-source sleep corpus yields OSF, which generalizes better to missing channels and scales with data and model size.","lead":"Researchers built a large open dataset of sleep recordings and tested how different self-supervised pretraining methods perform on sleep staging and disease prediction. They found that training models to ignore which channels are present makes them more robust when channels are missing, and that their OSF models beat existing sleep foundation models on most benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Finding 3's multi-source advantage is confounded with sample size: at matched 'scale' percentages, multi-source has ~2.7x more epochs than SHHS-only, so diversity benefits are not isolated from data quantity.","rationale":"The reader's weakest assumption (zero-padding as a proxy for absent sensors) is a genuine external-validity concern, and the suggested validation on real partial-channel recordings would still be valuable. However, the most load-bearing weakness in the central claim is the multi-source scaling experiment: the comparison in Sec. 5 and D.2 uses different numbers of pre-training epochs for single-source vs. multi-source at every reported scale. Since the paper itself demonstrates performance scaling with sample size, the multi-source benefit cannot be attributed to diversity. This internal confound is more decisive than the zero-padding issue because it invalidates a headline finding even if all simulation assumptions are accepted. The appropriate verdict remains CONDITIONAL (the reader's verdict), but the conditions should include re-running the multi-source comparison with matched epoch counts or removing the diversity interpretation. Thus I leave the verdict unchanged, while flagging the need for this additional control.","tokens_in":23978,"tokens_out":16420,"duration_ms":163489,"concrete_test":"Re-run the single-source vs. multi-source comparison with equal total pre-training epochs. Concretely, train OSF on (a) SHHS-100% (5,919k epochs), (b) a random 5,919k-epoch subsample of the full multi-source corpus (stratified across datasets), and repeat at 1% and 10% scaled epoch counts, keeping all hyperparameters identical and evaluating linear-probing on MROS and SHHS. If (b) does not consistently beat (a) across tasks and scales, Takeaway 1 / Finding 3's diversity claim should be withdrawn or reframed as a data-quantity effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Finding 3 and Takeaway 1 claim that multi-source data mixture improves downstream generalization (Sec. 5; detailed in D.2). The supporting comparison does not hold pre-training sample size fixed. For each scale (1%, 10%, 100%), the SHHS-only condition uses only SHHS epochs (5.9M at 100%), while the multi-source condition uses the full nine-dataset corpus (16.0M at 100%, Table 14). Because the paper's own Finding 3 shows that downstream performance rises with pre-training sample size, the multi-source advantage is fully confounded with total epoch count. The text says the variants are trained 'under identical settings,' but identical percentage of each cohort is not identical sample size. Thus the diversity component of the recipe is unsupported unless the number of pre-training epochs is matched. This is an internal validity problem, not an external assumption: the control condition is missing. The zero-padding concern raised by the reader is valid, but the multi-source confound is more load-bearing because it directly undermines one of the three headline findings without requiring any assumptions about deployment realism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SleepBench, a large open benchmark of 166,500 hours of polysomnography from nine public datasets, and uses it to systematically compare four families of self-supervised pre-training objectives (contrastive, reconstruction, autoregressive, distillation) for sleep foundation models. Three findings are claimed: (1) existing sleep FMs degrade sharply when channels are missing at inference; (2) channel-masking augmentation, which encourages channel-invariant representations, substantially improves downstream transfer for invariance-based methods; and (3) scaling pre-training data, model capacity, and multi-source data mixture consistently improves performance. The authors then propose OSF, a DINO-based model with a two-stage time/channel masking pre-training recipe, and report state-of-the-art results across sleep staging, sleep event detection, few-shot adaptation, and patient-level disease prediction on in-domain and out-of-domain cohorts.","tokens_in":24305,"tokens_out":6432,"duration_ms":55611,"significance":"If the findings hold, this is a valuable contribution: it provides the largest fully open sleep benchmark to date, a controlled comparison of SSL design choices in a domain where such analyses are scarce, and a concrete pre-training recipe (channel masking plus self-distillation) that appears to improve transfer and robustness. The strengths include the scale and openness of the corpus, the use of held-out external cohorts (e.g., MROS), and detailed ablations. However, the load-bearing claim that multi-source data diversity helps is confounded with total pre-training sample size, and the missing-channel evaluation relies on zero-padding as a proxy for absent sensors; both issues affect the central findings. The lack of error bars further weakens the quantitative SOTA comparisons. With these points addressed, the paper would be a strong candidate for acceptance.","major_comments":[{"comment":"The multi-source versus single-source comparison does not hold pre-training sample size fixed. Single-source is SHHS-only (5.9M train epochs at 100%, Table 14), while multi-source uses the full corpus (16.0M train epochs at 100%). At each data percentage (1%, 10%, 100%), multi-source uses ~2.7x more epochs. Since the paper's own Finding 3 (Sec. 3.3, Fig. 5) shows that downstream performance improves with pre-training sample size, the multi-source advantage in Tables 26-29 and Fig. 8 is fully confounded with total epoch count. The text in D.2 says the variants are trained 'under identical settings,' but identical percentage of each cohort is not identical sample size. Therefore, the diversity component of the recipe is unsupported. Please add a matched-sample-size control, e.g., subsample the multi-source corpus to the SHHS epoch count, or train SHHS-only with repeated epochs, to isolate","section":"Sec. 5 (Takeaway 1), Appendix D.2, Tables 26-29"},{"comment":"All missing-channel evaluation uses zero-padded or zero-masked channels as a proxy for absent sensors. The model may exploit zero-pattern artifacts (constant-value channels, padding positions) rather than learn true channel-invariant representations, and real missing channels also remove channel identity/calibration information that the architecture may still receive. The claims 'existing sleep FMs fail to generalize under missing-channel inference' (Finding 1), 'channel-invariant feature learning is essential' (Finding 2), and that OSF 'better addresses practical constraints' (Sec. 6.1) rest on this simulation. Please provide evidence on naturally missing-channel cohorts (e.g., datasets that natively lack certain channels) or analyze sensitivity to the zero-padding implementation, such as random channel dropout at inference or a learned mask embedding. The paper's acknowledgment that 'a","section":"Sec. 2, Sec. 3.1, Sec. 6.1 (Tables 5-6)"},{"comment":"No error bars or multiple seeds are reported. Many decisive comparisons are small: e.g., Table 2 fine-tuning hypopnea AUC (OSF 85.0 vs MAE 85.1), Table 4 diabetes AUPRC (OSF 56.3 vs VQ-VAE 54.6), and Table 6 missing-channel gains often under 1.0 AUC. Since each method was independently tuned to its 'best-performing regime' (App. A.1), it is unclear whether the reported gaps exceed run-to-run noise. Please report mean ± std over at least 3 seeds, or provide paired significance tests for the main OSF-versus-baseline comparisons. Without this, the state-of-the-art claim is not quantitatively grounded.","section":"All main tables (e.g., Tables 1-6, 26-32)"}],"minor_comments":[{"comment":"The masking-ratio specification 'r∈0.3,0.6' should be written as r ~ Uniform[0.3, 0.6]; the interval notation is ambiguous.","section":"Sec. 3.2"},{"comment":"The header row 'AROSF(SimCLR)OSF(DINO)' appears to collapse multiple columns; it should be 'AR | OSF(SimCLR) | OSF(DINO)' or similar.","section":"Table 10"},{"comment":"Typographical errors: 'Somantic' should be 'Somatic' and 'Nassal Pressure' should be 'Nasal Pressure'.","section":"Table 17"},{"comment":"The method 'MTAE' appears in few-shot results but is never defined in the main text or appendix; if this is a typo for 'MAE', please correct it, otherwise define it.","section":"Tables 20-22"},{"comment":"The row label 'In-Domain Datasets Out-of-Domain Datasets' is confusing; the table would benefit from a clear separator or a caption indicating which columns correspond to in-domain vs OOD cohorts.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The multi-source confound is the most serious issue because it directly undermines one of the three headline findings. It is, however, fixable with a matched-sample-size experiment. The zero-padding concern is also important but may be addressable with sensitivity analyses and a more careful discussion of limitations. The paper's open benchmark and code are strong assets; with the confounding resolved and basic uncertainty quantification added, this would likely be a well-cited contribution. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the open benchmark (SleepBench, 166.5k hours) and the controlled comparison of four SSL families are genuinely useful contributions; the channel-masking augmentation finding is the real meat and it holds up in the ablations. Second, the paper's Finding 3 overclaims on multi-source pre-training: the single-source vs multi-source comparison is run at the same percentage of each corpus, not the same number of epochs. At 100%, single-source has ~5.9M epochs (SHHS) and multi-source has ~16.0M (all nine datasets). Since the paper itself shows downstream performance rises with pretraining sample size, the multi-source advantage is almost entirely explained by data quantity, not diversity. Worse, the paper's own tables at 100% show no consistent multi-source advantage: in Table 26, single-src equals or beats multi-src on arousal, hypopnea, and ox. desat. on SHHS. Yet the text in D.2 claims 'consistently outperforms.' That's an internal inconsistency and it needs to be addressed before the claim stands.\n\nThe best parts: SleepBench is a real resource, the four-family comparison is the broadest controlled study I've seen for sleep FMs, and the channel-masking recipe is convincingly supported across SimCLR and DINO, with large, consistent gains. The scaling with model size and with sample size within a fixed cohort also looks real. Credit where due: the paper ships code/checkpoints, and the ablations are extensive.\n\nSoft spots worth naming. (1) The multi-source confound above is load-bearing, not cosmetic. (2) All missing-channel conclusions rely on zero-padding channels at inference. That treats absent sensors as zero-valued present sensors; real device dropouts also remove channel identity and calibration. The paper's robustness claims in Sec 6.1 may not transfer until tested on real partial-channel recordings. (3) No error bars or seeds, and each method is tuned to its 'best-performing regime' (App A.1). Differences like 97.3 vs 97.0 are within noise for all we know. (4) 'State-of-the-art' is overstated in places: on central apnea full fine-tuning, OSF does not lead in AUPRC (Table 2).\n\nVerdict: this is a serious empirical paper that deserves peer review. The benchmark and channel-masking insight are valuable regardless of the multi-source claim. But the multi-source finding needs either a matched-epoch comparison or a toned-down conclusion, and the missing-channel evaluation needs a real-data check. If those are fixed, this becomes a strong reference for sleep/physiological FM work.\n\nI'd bring it to a reading group, and I'd cite the benchmark and the channel-masking result.","headline":"The benchmark and channel-masking result are solid, but the multi-source scaling claim is confounded by sample size; worth refereeing with that fixed.","tokens_in":24740,"tokens_out":2937,"would_cite":true,"duration_ms":28261,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Channel masking during pre-training makes sleep foundation models robust to missing sensors.","keywords":["sleep foundation models","polysomnography","self-supervised pre-training","channel invariance","missing-channel robustness","scaling laws","multi-source data","disease prediction"],"falsifier":"Evaluate an OSF checkpoint on genuine recordings that never collected a particular sensor (for example, a home study without EEG) and compare performance to the paper's zero-masking simulation on full PSG recordings; if the real missing-sensor gap is much larger than the simulated gap, the channel-invariance conclusion does not transfer to practice.","tokens_in":23913,"feed_emoji":"🌙","tokens_out":6936,"duration_ms":66762,"temperature":0.7,"pith_summary":"This paper tries to establish that the pre-training recipe determines whether a sleep foundation model generalizes to real clinical conditions, where sensor sets differ and channels drop out. Evaluating four families of self-supervised objectives on a 166,500-hour corpus of polysomnography (multi-signal sleep recordings) from nine public datasets, it finds that existing models lose substantial accuracy when brain or respiratory channels are missing at inference, and that pre-training with random 50% channel masking—zeroing out half of the 12 standardized channels—restores robustness and improves transfer. With that recipe, models keep improving as pre-training data, model capacity, and the number of data sources increase, and the authors package the findings into OSF, which reports top performance across sleep staging, sleep-event detection, and patient-level disease prediction on all nine cohorts. The paper also acknowledges that a meaningful gap remains when task-critical channels are entirely absent, so the recipe narrows rather than eliminates the missing-channel problem.","feed_headline":"Zero out half the channels to make sleep models robust","feed_subtitle":"A pre-training trick that handles missing brain or breathing sensors, and keeps improving with more data.","key_machinery":"The central object is OSF, a pre-training recipe built on self-distillation with a two-stage masking augmentation: randomly zero 50% of the input channels, then apply contiguous temporal block masking to the remaining signals. A Transformer encoder is trained on 30-second epochs, resampled to 64 Hz from a standardized 12-channel montage (EEG, EOG, respiratory effort, airflow, snore, ECG, and three EMG leads). The channel-masking step carries the argument: by making two views of the same epoch differ in which physiological signals are present, the encoder cannot rely on sensor identity and must learn features aligned with physiology. For patient-level tasks, a top-k selection module aggregate","core_discovery":"At the center is a negative result and a fix. The negative result: current sleep foundation models are not reliable when inference-time inputs lack channels they saw during pre-training—removing brain channels hurts sleep staging, removing respiratory channels hurts hypopnea detection. The fix: explicitly force channel-invariant feature learning during pre-training by randomly zeroing 50% of input channels and requiring two augmented views to align. The authors show this works for both contrastive and self-distillation objectives, that it turns data scaling from a saturating curve into a consistently improving one, and that the resulting OSF family yields the strongest representations in lin","pith_inferences":["The missing-channel story is tested by zero-masking recorded signals; a direct comparison on recordings where sensors were physically absent would separate true channel invariance from learned zero-pattern shortcuts, and is the natural next experiment.","If channel dropout is the mechanism, the same augmentation should transfer to other physiological time-series models—wearable ECG, respiratory bands, home monitors—where sensor absence is common; the paper's recipe is directly testable there.","The top-k aggregation finding suggests disease-relevant information concentrates in specific sleep epochs; a clinical follow-up could check whether the selected epochs correspond to interpretable stages or events like apnea or arousal.","The scaling curves are demonstrated up to 85M parameters and roughly 166,500 hours; whether they flatten at substantially larger scale—or whether data-hungry objectives like OSF's will dominate—are questions the paper leaves open."],"forward_implications":["Sleep models built this way will tolerate realistic deployment settings—headband-only EEG studies, home studies without brain signals, airflow-only setups—with much smaller accuracy drops than current models.","Pooling many public sleep cohorts is worth the distribution shift: multi-source pre-training improves out-of-domain generalization at every data scale tested.","Scaling continues to pay off: larger transformers and more pre-training hours turn into better linear-probing, fine-tuning, and few-shot performance, giving a concrete roadmap to stronger sleep models.","Rare sleep-event labels can be exploited more efficiently; OSF is consistently better with 1, 5, and 50 labeled examples per class, which matters for tasks with scarce annotations.","Patient-level disease prediction should select informative epochs rather than average the whole night; a top-k selector outperforms mean pooling and trainable sequence aggregators in the paper's comparisons."],"fun_headline_variants":["Sleep AI stays accurate when sensors fail","Randomly dropping half of channels during training fixes sleep models","Channel-invariant pre-training solves the missing-sensor problem in sleep AI","OSF: sleep foundation models that scale and tolerate missing channels","For sleep models, zeroing 50% of channels at pretraining improves robustness"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The robustness and deployment claims rest on treating zero-masked or zero-padded channels as a faithful stand-in for sensors that were never recorded, even though real dropouts also remove channel identity and calibration information.","fun_headline_variants_meta":{"raw":{"variants":["Sleep AI stays accurate when sensors fail","Randomly dropping half of channels during training fixes sleep models","Channel-invariant pre-training solves the missing-sensor problem in sleep AI","OSF: sleep foundation models that scale and tolerate missing channels","For sleep models, zeroing 50% of channels at pretraining improves robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1669,"prompt_tokens":717,"completion_tokens":952,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":866}},"tokens_in":461,"tokens_out":952,"duration_ms":10099,"temperature":1.0,"reasoning_tokens":866,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T20:14:38.965579+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate an OSF checkpoint on genuine recordings that never collected a particular sensor (for example, a home study without EEG) and compare performance to the paper's zero-masking simulation on full PSG recordings; if the real missing-sensor gap is much larger than the simulated gap, the channel-invariance conclusion does not transfer to practice.","supporting_citations":[],"review_version":1}