REVIEW 4 major objections 5 minor 42 references
StableSleep: Source-Free Test-Time Adaptation for Sleep Staging with Lightweight Safety Rails
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A lightweight recipe adapts sleep-staging models to new patients in real time, using only unlabeled EEG and no source data.
desk verdict A sensible TTA recipe for sleep staging whose central claim—gains over a frozen baseline—is never actually measured. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the two-rail adaptation loop wrapped around Tent. On each streaming micro-batch the model minimizes the entropy of its softmax output while touching only batch-norm affine parameters and running statistics; an entropy gate keeps an exponential-moving-average (EMA) of batch entropy and suppresses updates outside [hmin, hmax], so near-uniform or artefactual windows cannot corrupt the weights; an EMA snapshot of the adapted parameters acts as the reset anchor, restoring the model when a drift criterion fires. A causal median filter of width 5 smooths prediction flicker. The gate and reset are the load-bearing additions: they turn a memoryless entropy minimizer into a process th
What would settle it
Deploy the exact recipe, with validation-fixed hyperparameters untouched, on a different public sleep dataset (different montage or population) and compare the adapted model against the same frozen baseline; if the gains flip sign on the first external cohort, the claim of consistent deployment-ready gains is refuted. A cheaper check inside the corpus: re-split subjects and re-tune the gate bounds on the new validation set — if the optimal [hmin, hmax] moves with the split, the settings are not transferable.
Extended reading notes
Core claim
This is an engineering claim about what makes test-time adaptation safe in a clinical stream: entropy minimization alone is not enough, and the contribution is the wrapper that stabilizes it. A source-trained model with explicit batch-norm layers adapts on a new patient's stream by minimizing prediction entropy per micro-batch, touching only batch-norm affine parameters and running statistics, skipping updates when the EMA of batch entropy leaves [hmin, hmax], and resetting to an EMA snapshot when drift is detected. On Sleep-EDF Expanded (Fpz–Cz, 100 Hz, 30 s epochs; subject-disjoint; settings fixed from validation; test labels unused), the model reaches 67.0% accuracy and κ = 0.394 — stated
Load-bearing premise
The whole result rests on the assumption that the adaptation settings fixed on the validation subjects — the entropy-gate bounds, the reset trigger, the batch-norm momentum, the smoothing width — carry over unchanged to any new patient; the paper gives no sensitivity analysis, so if those thresholds are patient-specific, the reported consistency could be an artifact of the chosen validation split.
Editorial extensions
If this is right
- Frozen staging models can be fielded on a new patient with no retraining, no source data, and no labels — the model adapts itself from the incoming EEG stream.
- Adaptation runs at seconds-level latency with minimal memory, so it fits on laptop-class or bedside hardware rather than research clusters.
- Because only batch-norm layers change, the wrapper is model-agnostic: any staging backbone with explicit batch-norm can carry the same rails.
- Privacy is satisfied by construction: no patient labels and no source data ever enter the adaptation loop.
- The gain profile is uneven by stage — N1 remains hardest with dominant N2/REM confusions — so deployments should still expect transitional epochs to be the weak spot.
Reading between the lines
- The two rails use nothing sleep-specific, so the same wrapper should transfer to other streaming physiological classifiers (seizure detection, cardiac rhythm monitoring), where artefact windows and slow drift are the same failure modes — a connection the paper does not draw.
- The absolute gate bounds [hmin, hmax] are a tuning liability; a gate defined relative to the running entropy distribution could plausibly remove the validation-tuning step, a testable variant the paper does not explore.
- Because calibration stays at ECE ≈ 0.08, entropy minimization does not automatically buy trustworthy confidence; adding confidence-based deferral or post-hoc recalibration could convert the accuracy gains into clinically usable uncertainty.
- The untested edge of the claim is external-cohort transfer: the recipe would be most valuable exactly where it is hardest, moving from Sleep-EDF to a different montage and population.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes StableSleep, a streaming, source-free test-time adaptation (TTA) recipe for sleep staging. It combines BatchNorm (BN) statistic refresh with entropy minimization (Tent) and adds two safety rails: an entropy gate that pauses adaptation on uncertain windows and an EMA-based reset that reverts to a snapshot when drift is detected, followed by a causal median filter. The method is evaluated on Sleep-EDF Expanded using single-lead EEG (Fpz-Cz), with subject-disjoint splits and no test-label access. The authors claim consistent gains over a frozen baseline at seconds-level latency and minimal memory. The central weakness is that these claims are not supported by the reported evidence: Table 1 reports only adapted-model aggregate metrics, Section 4.5 is a qualitative summary, Sections 4.3 and 4.4 are explicit placeholders, and no frozen-baseline numbers or hyperparameter values are given.
Significance. If the headline claim were substantiated, StableSleep would be a practical and useful contribution: it is simple, privacy-preserving (no source data or patient calibration), and plausibly suitable for on-device use. The evaluation protocol has appropriate elements: subject-disjoint splits, a streaming constraint, and no test-label access. However, as presented, the empirical support is incomplete. The absolute metrics (test accuracy 67.0%, macro-F1 35.1%, Cohen's kappa 0.394) cannot be interpreted without a frozen-baseline comparison or numeric ablations. The paper provides no machine-checked proofs and no code release; reproducibility is promised only upon acceptance.
major comments (4)
- [Section 3.4, Table 1, Section 5] The abstract and Section 5 state 'consistent gains over a frozen baseline,' but no frozen-baseline metrics are reported anywhere. Table 1 shows only adapted-model aggregate metrics for validation and test; there is no baseline column, no per-subject deltas, no BN-only numeric comparison, and no Tent-without-gates comparison. Section 4.5 asserts that 'BN-only improves over frozen inference' and 'Tent adds further gains' without presenting any numbers. Because the central contribution is the improvement over the frozen model, these comparisons must be reported, including per-subject variability.
- [Section 4.5, Sections 4.3/4.4] The ablation study is described as a 'qualitative summary' with no numerical results, standard deviations, or paired significance tests. Sections 4.3 and 4.4 are explicitly placeholders ('added back'), so the calibration and stage-distribution analyses are not actually provided. This is more than a presentation issue: the safety rails (entropy gate, EMA reset) are core contributions, and the paper needs quantitative ablations (gate on/off, reset on/off, BN-only vs. Tent) plus subject-level spread to support the claimed benefits.
- [Section 3.4] The adaptation hyperparameters — entropy gate thresholds hmin/hmax, EMA reset criterion and momentum, BN running-statistics momentum, and causal median filter width — are defined but no values are given. Section 4.5 says these are fixed from validation, but no sensitivity analysis is reported. If the thresholds are validation-specific, the claimed consistency on test may not transfer; the authors should report the chosen values and a sensitivity sweep over them.
- [Section 5] The abstract and conclusion promise 'seconds-level latency and minimal memory,' but no inference latency or memory measurements are reported. Appendix B.1 gives training runtime only. Since deployment efficiency is part of the claimed contribution, the paper should include profiling results on the target hardware or at least characterize per-epoch computational cost.
minor comments (5)
- [Sections 4.3/4.4] The headings 'Calibration (added back)' and 'Stage distribution (added back)' are placeholders; they should either be completed with actual analyses and figures or removed.
- [References] Several references are duplicated or incomplete: [1]/[2], [3]/[4], [15]/[16], and [17]/[18] appear to be near-duplicates; [5] lacks a full author list; [32] is listed but not cited in the text. Please consolidate and check all citations.
- [Section 3.3] The phrase 'model-agnostic' is overstated if adaptation relies on BN layers; specify the class of architectures for which the recipe applies.
- [Appendix A] Figure 8 is referenced for subject-wise distributions, but numeric standard deviations or interquartile ranges are not reported. Adding these would strengthen the claims about consistency.
- [Section 3.2] Please clarify whether per-record streaming standardization is updated before or after TTA updates affect the BN statistics; this affects the online/streaming claim.
Circularity Check
No circular derivation found; headline performance claim is unquantified but not circular.
full rationale
This paper is an empirical evaluation of a test-time adaptation recipe rather than a derivation. The method section defines the entropy-minimization objective, BN refresh, entropy gate, and EMA reset without fitting any target quantity into the definition. Table 1 reports only adapted-model metrics; the abstract’s claim of 'consistent gains over a frozen baseline' is asserted in Section 4.5 ('BN-only improves over frozen inference', 'Tent adds further gains') but no frozen-baseline numbers are provided anywhere. That is a missing-comparison / evidence problem, not circularity: no equation reduces to its own input, no fitted parameter is renamed as a prediction, and no load-bearing self-citations appear (all references are standard prior work by others). Sections 4.3 and 4.4 are placeholder descriptions ('added back') rather than quantitative analyses, further indicating incompleteness rather than circular reduction. Hyperparameters are selected on validation and frozen for test, which is standard practice and not circular. Therefore no circular step can be exhibited with a quote-and-reduction, and the correct finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- Entropy gate thresholds (hmin, hmax)
- EMA reset criterion and momentum
- BatchNorm running statistics momentum
- Causal median filter width =
5
assumptions (3)
- domain assumption Entropy minimization is a valid proxy for correctness on unlabeled target streams
- domain assumption The R&K to AASM mapping preserves label semantics for this dataset
- domain assumption The domain shift in Sleep-EDF Expanded mainly affects feature statistics rather than the feature extractor
Cite this review
Pith. "Pith review of StableSleep: Source-Free Test-Time Adaptation for Sleep Staging with Lightweight Safety Rails." pith.science (2026). https://pith.science/paper/RIAROQVV
@misc{pith2026250902982,
author = {Pith},
title = {Pith review of: StableSleep: Source-Free Test-Time Adaptation for Sleep Staging with Lightweight Safety Rails},
year = {2026},
howpublished = {\url{https://pith.science/paper/RIAROQVV}},
note = {Machine review of arXiv:2509.02982}
}
read the original abstract
Sleep staging models often degrade when deployed on patients with unseen physiology or recording conditions. We propose a streaming, source-free test-time adaptation (TTA) recipe that combines entropy minimization (Tent) with Batch-Norm statistic refresh and two safety rails: an entropy gate to pause adaptation on uncertain windows and an EMA-based reset to reel back drift. On Sleep-EDF Expanded, using single-lead EEG (Fpz-Cz, 100 Hz, 30s epochs; R&K to AASM mapping), we show consistent gains over a frozen baseline at seconds-level latency and minimal memory, reporting per-stage metrics and Cohen's k. The method is model-agnostic, requires no source data or patient calibration, and is practical for on-device or bedside use.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
- [1]
-
[2]
Berry, Rita Brooks, Charlene Gamaldo, Susan M
Richard B. Berry, Rita Brooks, Charlene Gamaldo, Susan M. Harding, Robert M. Lloyd, Carole L. Marcus, and Bradley V . Vaughn.The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications. American Academy of Sleep Medicine, Darien, IL, version 2.2 edition, 2015
work page 2015
-
[3]
Berry, Rohit Budhiraja, Daniel J
Richard B. Berry, Rohit Budhiraja, Daniel J. Gottlieb, and et al. Rules for scoring respiratory events in sleep: Update of the 2007 aasm manual. Journal of Clinical Sleep Medicine, 8(5):597– 619, 2012
work page 2007
-
[4]
Berry, Rohit Budhiraja, Daniel J
Richard B. Berry, Rohit Budhiraja, Daniel J. Gottlieb, David Gozal, Conrad Iber, Vishesh K. Kapur, Carole L. Marcus, Reena Mehra, Sairam Parthasarathy, Stuart F. Quan, Susan Redline, Kingman P. Strohl, Susan L. D. Ward, and Michael M. Tangredi. Rules for scoring respiratory events in sleep: update of the 2007 AASM Manual for the Scoring of Sleep and Assoc...
work page 2007
-
[5]
Richard B Berry et al. The aasm manual for the scoring of sleep and associated events: Rules, terminology and technical specifications (version 2.0). American Academy of Sleep Medicine, 2012
work page 2012
-
[6]
Stanislas Chambon, Mathieu N Galtier, Pierrick J Arnal, Gilles Wainrib, and Alexandre Gram- fort. A deep learning architecture for temporal sleep stage classification using multimodal and multivariate time series. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2018
work page 2018
-
[7]
A coefficient of agreement for nominal scales
Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960
1960
-
[8]
An attention-based deep learning approach for sleep stage classification with single-channel eeg
Emadeldeen Eldele, Zhenghua Chen, Chengyu Liu, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. An attention-based deep learning approach for sleep stage classification with single-channel eeg. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 29:809–818, 2021
work page 2021
Show all 42 references
-
[9]
Time-Series Representation Learning with Temporal Convolutional Networks and Attention for Sleep Stage Classification
Emadeldeen Eldele, Mohamed Ragab, Zhe Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-Series Representation Learning with Temporal Convolutional Networks and Attention for Sleep Stage Classification. IEEE Transactions on Neural Systems and Rehabilitation Engine...
2021
-
[10]
Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning. arXiv:1702.03118, 2017
2017 arXiv
-
[11]
Physiobank, physiotoolkit, and physionet: Components of a new research resource for complex physiologic signals
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: Components of a new research resource for complex physiologic si...
2000
-
[12]
Mne software for processing meg and eeg data
Alexandre Gramfort et al. Mne software for processing meg and eeg data. NeuroImage, 86:446–460, 2014
2014
-
[13]
Engemann, Daniel Strohmeier, Christian Brodbeck, Roman Goj, Mainak Jas, Teon Brooks, Lauri Parkkonen, and Matti S
Alexandre Gramfort, Martin Luessi, Eric Larson, Denis A. Engemann, Daniel Strohmeier, Christian Brodbeck, Roman Goj, Mainak Jas, Teon Brooks, Lauri Parkkonen, and Matti S. Hämäläinen. MEG and EEG Data Analysis with MNE-Python. Frontiers in Neuroscience, 7:267, 2013
2013
-
[14]
On calibration of modern neural networks
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning (ICML) , 2017. 5
2017
-
[16]
Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. In arXiv preprint arXiv:1704.04861, 2017
2017 arXiv
-
[17]
Squeeze-and-excitation networks
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
2018
-
[18]
Squeeze-and-Excitation Networks
Jie Hu, Li Shen, Gang Sun, Samuel Albanie, and Enhua Wu. Squeeze-and-Excitation Networks. In Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2018
2018
-
[19]
Chesson, and Stuart F
Conrad Iber, Sonia Ancoli-Israel, Andrew L. Chesson, and Stuart F. Quan. The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications. American Academy of Sleep Medicine, 2007
2007
-
[20]
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, 2015
2015
-
[21]
The Sleep-EDF Database
Bob Kemp. The Sleep-EDF Database. PhysioNet/PhysioBank, 2002
2002
-
[22]
Sleep-EDF Database Expanded
Bob Kemp, Ana Cristina da Rosa, Joost van Dijk, et al. Sleep-EDF Database Expanded. PhysioNet News, 2018
2018
-
[23]
Analysis of a sleep-dependent neuronal feedback loop: The slow-wave microcontinuity of the eeg
Bob Kemp, Aeilko H Zwinderman, Bauke Tuk, Henri A Kamphuisen, and Jeroen J Oberye. Analysis of a sleep-dependent neuronal feedback loop: The slow-wave microcontinuity of the eeg. IEEE Transactions on Biomedical Engineering, 2000
2000
-
[24]
Zwinderman, Bert Tuk, Hilbert A
Bob Kemp, Aeilko H. Zwinderman, Bert Tuk, Hilbert A. C. Kamphuisen, and Josefien J. L. Oberyé. Analysis of a sleep-dependent neuronal feedback loop: The slow-wave microcontinuity of the eeg. IEEE Transactions on Biomedical Engineering, 47(9):1185–1194, 2000
2000
-
[25]
ISRUC-sleep: A comprehensive public dataset for sleep researchers
Sirvan Khalighi, Teresa Sousa, José Moutinho Santos, and Urbano Nunes. ISRUC-sleep: A comprehensive public dataset for sleep researchers. Computer Methods and Programs in Biomedicine, 124:180–192, 2016
2016
-
[26]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR), 2015
2015
-
[27]
Revisiting batch normalization for practical domain adaptation
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04779, 2016
2016 arXiv
-
[28]
Adaptive batch normal- ization for practical domain adaptation
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Adaptive batch normal- ization for practical domain adaptation. Pattern Recognition, 80:109–117, 2018
2018
-
[29]
Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), 2020
2020
-
[30]
Focal Loss for Dense Object Detection
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. In Proc. IEEE International Conference on Computer Vision (ICCV), 2017
2017
-
[31]
Long-tail learning via logit adjustment
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In Advances in Neural Information Processing Systems, 2020
2020
-
[32]
Efficient test-time model adaptation without forgetting
Shuaicheng Niu, Jian Wang, Chang Ren, Gaofeng Zhang, Jinjin Liao, and Tiejun Huang. Efficient test-time model adaptation without forgetting. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022
2022
-
[33]
Montreal archive of sleep studies: An open-access resource for instrument benchmarking and exploratory research
Christian O’Reilly, Nadia Gosselin, Julie Carrier, and Tore Nielsen. Montreal archive of sleep studies: An open-access resource for instrument benchmarking and exploratory research. Journal of Sleep Research, 23(6):628–635, 2014. 6
2014
-
[34]
U-Sleep: Resilient high-frequency sleep staging
Mathias Perslev, Sune Darkner, Lykke Kempfner, Miki Nikolic, Poul Jørgen Jennum, and Christian Igel. U-Sleep: Resilient high-frequency sleep staging. npj Digital Medicine, 4(72), 2021
2021
-
[35]
U-time: A fully convolutional network for time series segmentation applied to sleep staging
Mathias Perslev, Michael Hejselbak Jensen, Sune Darkner, Poul Jørgen Jennum, and Christian Igel. U-time: A fully convolutional network for time series segmentation applied to sleep staging. In Advances in Neural Information Processing Systems (NeurIPS), 2019
2019
-
[36]
Chén, and Maarten De V os
Huy Phan, Fernando Andreotti, Navin Cooray, Oliver Y . Chén, and Maarten De V os. Seqsleepnet: End-to-end hierarchical recurrent neural network for sequence-to-sequence automatic sleep staging. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 27(3):400–410, 2019
2019
-
[37]
Quan, Barbara V
Stuart F. Quan, Barbara V . Howard, Conrad Iber, John P. Kiley, F. Javier Nieto, George T. O’Connor, David M. Rapoport, Susan Redline, John Robbins, Jonathan M. Samet, and Patri- cia W. Wahl. The sleep heart health study: Design, rationale, and methods. Sleep, 20(12):1077– 1085, 1997
1997
-
[38]
A Manual of Standardized Terminology, Techniques and Scoring System for Sleep Stages of Human Subjects
Allan Rechtschaffen and Anthony Kales. A Manual of Standardized Terminology, Techniques and Scoring System for Sleep Stages of Human Subjects . U.S. National Institutes of Health, 1968
1968
-
[39]
Efros, and Moritz Hardt
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020
2020
-
[40]
DeepSleepNet: A Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG
Akara Supratak, Hao Dong, Chao Wu, and Yike Guo. DeepSleepNet: A Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG. arXiv:1703.04046, 2017
2017 arXiv
-
[41]
Tent: Fully test-time adaptation by entropy minimization
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations (ICLR), 2021
2021
-
[42]
Continual test-time domain adaptation
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7201–7211, 2022
2022
-
[43]
Inter-database validation of a deep learning approach for automatic sleep staging
Diego Álvarez Estévez, Valentín Moret-Bonillo, Clara Lado, et al. Inter-database validation of a deep learning approach for automatic sleep staging. PLOS ONE, 16(8):e0256111, 2021. 7 A Supplementary figures Figure 4: Validation confusion matrix (counts and row-normalized). Fig...
2021
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.