{"id":"8be09b5f-8e9d-4c81-a422-49b4cbc5af63","arxiv_id":"2412.15412","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LG-Sleep reports 0.85 accuracy and 0.75 macro F1 for subject-independent three-class mouse sleep scoring from EEG, outperforming CNN, LSTM, FC, and three prior methods on the authors' dataset.","lead":"LG-Sleep is a deep neural network that scores mouse sleep stages from EEG by extracting short local patterns with a convolutional layer and longer global transitions with an LSTM, while an autoencoder-style reconstruction loss is meant to improve generalization. A generalist reader might care because it claims subject-independent automatic sleep scoring that still works when only a fraction of training labels are available.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Statistical support for the headline numbers is not established: repeated subject-out folds and per-window scoring ignore within-mouse autocorrelation, making the reported variances and t-test claims unreliable.","rationale":"The Reader's conditional verdict is appropriate, and my concern is a sharper version of their second structural assumption. The autoencoder latent-representation mechanism is also untested, but the evaluation-protocol issue is more load-bearing because it affects every number in Tables 2-5, including the significance claim. Even if the latent representation were ideal, the quantitative evidence for 'outperforms' would remain weak if the unit of analysis is the 1-second window rather than the mouse. The paper deserves credit for using subject-out splits, which is the right general design, but the repeated folds and per-window scoring do not provide the independent evidence needed for the reported standard deviations and p-value claims. Without code or per-mouse predictions, this reanalysis cannot be done from the paper alone, which is exactly why the conditional verdict should remain: acceptance should require the authors to supply per-mouse or per-independent-epoch results demonstrating that the reported performance gaps persist under subject-level inference.","tokens_in":7993,"tokens_out":9477,"duration_ms":93271,"concrete_test":"Recompute all headline comparisons with a subject-level protocol: (i) aggregate predictions per mouse before computing accuracy and macro-F1, yielding 16 per-mouse values per method; (ii) report leave-one-mouse-out (16-fold) results with per-mouse means and 95% bootstrap confidence intervals over mice; (iii) compare LG-Sleep to each baseline with a paired test on the 16 per-mouse accuracy differences, e.g., Wilcoxon signed-rank, instead of an unspecified t-test over many overlapping windows. If the per-mouse confidence intervals overlap between LG-Sleep and the CNN/LSTM baselines, or the paired comparison fails to reach p < 0.05, the claim that LG-Sleep outperforms existing algorithms is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LG-Sleep 'achieves high accuracy (0.85) and F1-score 0.75' and 'outperforms existing algorithms' rests entirely on the evaluation protocol in Section III. That protocol does not justify the reported uncertainty. With only 16 mice, a 4-fold subject-out split leaves only 4 test mice per fold, and 'repeating 4 times' does not add independent subjects. If metrics are computed over 1-second windows derived from 10-second trials (Section II-B, 500-ms overlap), adjacent windows are strongly non-independent, and even non-overlapping 10-second trials from the same mouse are highly autocorrelated because sleep-stage bouts persist across many trials. The table entries are therefore averaged over many correlated samples while the reported standard deviations and the unspecified t-test appear to treat them as independent. The comparison to re-implemented baselines is consequently not auditable evidence of superiority: the apparent margins (0.85 vs 0.71-0.75) could shrink or disappear if predictions are first aggregated per mouse or per independent recording epoch. This is an evaluation-validity concern, not an internal contradiction; the method may work, but the current numbers do not yet establish it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LG-Sleep, a deep neural network for three-class mouse sleep scoring (wake, REM, NREM) from single-channel EEG. The architecture uses a time-distributed 1D CNN to extract local temporal transitions, an LSTM to model global transitions, and an autoencoder-decoder layout with a reconstruction loss added to the classification loss. Experiments on an internal dataset of 16 mice with subject-independent four-fold cross-validation report a wide-band accuracy of 0.85 and macro F1-score of 0.75, and the paper claims superiority over CNN, LSTM, FC, and three re-implemented prior methods, as well as robustness when only 25--50% of labeled training data are used.","tokens_in":8256,"tokens_out":2976,"duration_ms":27585,"significance":"If the reported results are valid, LG-Sleep would be a useful baseline for automated mouse sleep scoring, particularly because it addresses class imbalance and limited labeled data, which are practical concerns in preclinical sleep research. The paper has several strengths: the architecture is clearly described, frequency-subband and ablation analyses are included, limited-data experiments are reported, and the limitations paragraph candidly acknowledges the REM detection difficulty and training cost. However, the central quantitative claims rest on an internal dataset with no shared code or data, re-implemented baselines, and an evaluation protocol that does not account for within-mouse temporal dependence; these issues currently weaken the evidence for the headline accuracy and superiority claims.","major_comments":[{"comment":"The evaluation protocol does not justify the reported uncertainties. With 16 mice, a 4-fold subject-out split yields only 4 test mice per fold, and 'repeating 4 times' does not create new independent subjects; the effective between-subject sample size remains 16. In addition, Section II-B describes 1-second windows with 500-ms overlap derived from 10-second trials, so adjacent windows are strongly non-independent, and even non-overlapping trials from the same mouse are autocorrelated because sleep stages persist across many trials. The reported standard deviations and the t-test mentioned in Section III-3 therefore appear to treat correlated windows as independent samples, which will underestimate variance and can make small performance differences appear significant. The authors should aggregate predictions per mouse (or per independent recording epoch) and use cluster-level statistics, and should report the number of independent units used in the t-test.","section":"Section III (Experiments and Discussion)"},{"comment":"The comparison with existing methods is not auditable because all baselines were 're-implemented in Python using TensorFlow based on the provided descriptions' rather than taken from official releases or code. Since the paper does not provide the re-implementation code, a reader cannot verify that the baselines were given fair hyperparameter treatment, and the claimed margins (0.85 vs 0.68--0.71 accuracy) could be affected by implementation details. To support the superiority claim, the authors should release and run official implementations where available, or provide detailed per-method hyperparameter search and code for the re-implementations.","section":"Table 5 and Section III-3"},{"comment":"The REM recall for LG-Sleep is 0.57, meaning 43% of REM windows are missed. Because REM comprises only 8% of the dataset, the macro F1 of 0.75 masks this weakness. The abstract and conclusion state that LG-Sleep 'performs well across different sleep stages' and 'achieves good performance across different sleep stages,' which is overstated relative to the confusion matrix. Per-class precision and recall should be reported and discussed, especially since the limitations paragraph itself acknowledges 'difficulties in detecting REM due to the limited number of samples.'","section":"Fig. 3 and Section III-1"},{"comment":"The paper asserts that the autoencoder reconstruction loss, with the latent vector repeated n times before the decoder LSTM, makes the learned representation stage-relevant and subject-invariant, but this mechanism is never directly tested. Specifically, there is no analysis showing that the latent space separates sleep stages independently of mouse identity, or that reconstruction primarily encodes stage-related signal rather than subject-specific noise. Without such evidence, the claimed generalization advantage over a plain CNN+LSTM remains plausible but unsupported. The authors could add latent-space visualization or a quantitative analysis of reconstruction errors across subjects and stages.","section":"Section II-C and Section III"}],"minor_comments":[{"comment":"There are several typographical issues, including 'JUL Y 2017' in the header, 'LG-SLeep' in the Table 5 caption, and inconsistent spacing throughout the text.","section":"General formatting"},{"comment":"Table 1 lists the decoder LSTM as 'LSTM (512; 0.2)' while Section II-C states the decoder LSTM has a hidden state size of 256. Please clarify which value is correct and ensure the table and text are consistent.","section":"Table 1 and Section II-C"},{"comment":"The slicing details are inconsistent: Section II-D says each sample size is 19×512×1, but the implementation section says the encoder output vector is repeated 18 times before the decoder LSTM. With a 10-second trial, 1-second windows, and 500-ms overlap, one would expect 19 windows; the relationship between the 19 slices and the 18 repeats should be explained or corrected.","section":"Sections II-B and II-D"},{"comment":"The caption states the background colors purple, red, and green correspond to wake, NREM, and REM, but the text in Section III describes a sample of true versus predicted labels without naming which colors correspond to which stages in subfigure (a) and (b). Please make the mapping explicit and consistent between text and figure.","section":"Fig. 2 caption"},{"comment":"The t-test is mentioned but no details are given about whether it is paired, how many observations are used, or whether it accounts for the repeated folds. Please specify the test, the unit of analysis, and the number of independent samples.","section":"Section III-3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's applied machine-learning scope, but the evaluation-validity issue is load-bearing: the headline accuracy and the significance tests are based on an analysis that treats dependent windows as independent. This is fixable by re-analyzing at the per-mouse level, but it requires re-running the experiments. The lack of shared code and data further reduces the verifiability of the comparison. I would encourage the editor to require the revised manuscript to include code or a detailed reproducibility description, and to address the statistical protocol before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The central claim of LG-Sleep — that it outperforms existing algorithms for subject-independent mouse sleep scoring — is plausible but not proven by the evidence as presented. The architecture itself is a reasonable, incremental design: time-distributed CNNs capture local transitions, an LSTM captures global context, and an autoencoder reconstruction loss is intended to encourage subject-invariant features. That is a sensible idea, and the ablation study shows that the full model beats its CNN, LSTM, and fully connected components on this private dataset. The limited-labeled-data results are also interesting: the model degrades gracefully at 50% labeled data, which would be practically useful if it holds up.\n\nThe soft spots are mostly in the evaluation. Sixteen mice is a small sample; a 4-fold subject-out split leaves only 4 test mice per fold, and repeating the splits four times does not create new independent subjects. The 500-ms overlapping windows mean adjacent samples are strongly correlated, and sleep-stage bouts make even non-overlapping 10-second trials from the same mouse non-independent. The reported standard deviations and the t-test appear to treat samples as independent, so the stated margins (0.85 vs 0.71–0.75) could shrink or vanish under per-mouse or per-trial aggregation. The confusion matrix shows REM recall of only 0.57; the authors acknowledge this, but it means the macro-F1 is pulled up by the dominant NREM class. On top of that, no code or data are released, and the baselines were re-implemented from textual descriptions, so the comparison cannot be audited.\n\nThe paper is honest about its limitations and the methodology is coherent; the load-bearing assumption that the reconstruction loss yields sleep-relevant, subject-invariant features is asserted but not directly tested, but that is a fixable gap rather than a fatal flaw. This is the kind of work that could become a useful baseline for the mouse sleep-scoring community if the evaluation were made statistically sound and the code/data were shared.\n\nIf it came to my desk, I would send it out for review — it deserves a serious referee — but I would ask for major revisions on the statistical analysis, per-subject aggregation, and data availability before accepting it. I would not cite it as a benchmark in my own work yet, though I might reference it as related work once the evaluation is cleaned up.","headline":"Plausible incremental architecture for mice sleep scoring, but the reported margin over baselines is not statistically established.","tokens_in":8779,"tokens_out":2510,"would_cite":false,"duration_ms":22098,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LG-Sleep claims an autoencoder-regularized CNN-LSTM scores 3-class mouse sleep from EEG at 0.85 accuracy and 0.75 macro F1, beating CNN, LSTM, FC, and three prior methods on subject-independent folds.","keywords":["sleep scoring","mouse EEG","temporal dependencies","CNN","LSTM","autoencoder-decoder","subject-independent classification","limited labeled data"],"falsifier":"Take the 32-dimensional latent vectors of held-out mice and color them by mouse identity: if a human or a simple classifier can separate subjects more cleanly than wake/NREM/REM stages, then the autoencoder is capturing subject-specific signal, and the claimed subject-independent generalization would not hold.","tokens_in":7790,"feed_emoji":"🐭","tokens_out":4972,"duration_ms":42226,"temperature":0.7,"pith_summary":"This paper claims that a deep network combining time-distributed convolutions and an LSTM, trained jointly with a reconstruction loss in an autoencoder-decoder setup, can score three mouse sleep stages (wake, NREM, REM) from EEG in a subject-independent way. On 16 mice with four-fold subject-out cross-validation, the network reports wide-band accuracy of 0.85 and macro F1 of 0.75, compared with lower accuracy and much lower F1 for CNN, LSTM, fully connected, and three re-implemented prior sleep scorers. It also reports that performance degrades gracefully when only 25-50% of labeled training samples are used. A sympathetic reader would take the central message to be that explicitly modeling both short local transitions (500 ms kernel slices) and long global sequence structure pays off, and that the reconstruction loss helps the learned features transfer across mice and tolerate scarce labels.","feed_headline":"EEG model scores mouse sleep at 85% accuracy and 0.75 F1","feed_subtitle":"A subject-independent network with an autoencoder objective beats CNN, LSTM, and prior sleep scorers on 3-class mouse EEG.","key_machinery":"The carrying mechanism is the paired encoder-decoder with a shared latent vector. The encoder uses time-distributed 1D CNNs (64 filters, kernel width 500 ms), max pooling, and an LSTM with hidden state size 32 to produce a flat latent representation, which is fed to a fully connected layer and softmax classifier $\\mathrm{softmax}(W A + b)$. The decoder repeats that latent vector 18 times, passes it through an LSTM of size 256, upsamples by a factor of 8, and applies two 1D CNN layers to reconstruct the input; the total loss is $\\mathcal{L} = \\mathcal{L}_c + \\mathcal{L}_{\\mathrm{mse}}$. The reconstruction constraint is what the paper argues forces the latent representation to retain enough information to rebuild the raw EEG while also being discriminative for sleep stages, producing cross-subject generalization and graceful degradation with limited labeled data.","core_discovery":"On its own terms, LG-Sleep establishes that a subject-independent mouse sleep scorer built from time-distributed 1D CNN filters over 500 ms EEG segments, an LSTM over the sequence, and an autoencoder-decoder reconstruction branch achieves mean accuracy 0.85 +/- 0.09 and macro F1 0.75 +/- 0.10 on wide-band EEG, outperforming CNN (0.73/0.57), LSTM (0.75/0.61), FC (0.66/0.48), and re-implementations of three published sleep scorers. The two-stage training first minimizes classification plus MSE reconstruction loss, then classification-only with class weights 1.5, 1, and 7 for wake, REM, and NREM, respectively; this is what the paper credits with robustness when training labels are limited. The paper also reports that delta, theta, alpha, and beta subbands perform similarly, while gamma drops to 0.62 accuracy and 0.29 F1.","pith_inferences":["Editorial inference: the same local/global encoder-decoder design could transfer to human sleep staging if the window length and kernel width are rescaled to human EEG rhythms, since the split into local and global temporal structure is not mouse-specific.","Editorial inference: the reconstruction branch is never probed directly, so a latent-space analysis (for example nearest-neighbor retrieval by stage) would reveal whether the autoencoder is the actual cause of cross-subject transfer or merely a regularizer.","Editorial inference: the 500 ms overlap between adjacent one-second slices means adjacent training trials share EEG content, so the reported fold standard deviations likely understate uncertainty; re-running with non-overlapping slices would give a more conservative reliability estimate."],"forward_implications":["A lab with EEG from a few mice could score wake/NREM/REM automatically at 0.85 accuracy without collecting per-subject labels, if the reported subject-out folds reflect realistic use.","The macro F1 of 0.75 shows that REM sleep is the limiting class, so scoring pipelines should focus data augmentation or cost weighting on REM rather than on overall accuracy.","Joint reconstruction-plus-classification training is a concrete regularization recipe that other EEG classifiers could adopt, with the second classification-only stage acting as fine-tuning.","Under annotation scarcity the drop is mild: 0.76 accuracy at 50% labeled data and 0.67 at 25%, making the architecture viable for low-label sleep studies."],"supporting_citations":[{"why":"Supplies the EEG dataset, surgery details, and the definitions of the three vigilance states used in the experiments.","marker":"[18]"},{"why":"A CNN-based automated mouse sleep scorer that is re-implemented as a baseline in Table 5 and must be outperformed.","marker":"[9]"},{"why":"A CNN+LSTM mouse sleep scorer (MC-SleepNet) that is re-implemented as a baseline and represents the prior combined architecture.","marker":"[10]"},{"why":"An LSTM-based pharmaco-EEG sleep scorer that is re-implemented as a baseline and represents the LSTM-only approach.","marker":"[11]"}],"fun_headline_variants":["LG-Sleep hits 85% accuracy on mouse sleep scoring","Subject-independent EEG net nails mouse sleep stages","Local-global temporal model beats CNN, LSTM on sleep","Autoencoder trick makes sleep scoring robust with few labels","Mice sleep scoring: LG-Sleep achieves 85% accuracy and 0.75 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the MSE reconstruction loss, together with the latent vector repeated 18 times before the decoder LSTM, forces the latent representation to encode sleep-stage information that transfers across individual mice; the paper does not directly test whether latent vectors separate by stage rather than by mouse identity.","fun_headline_variants_meta":{"raw":{"variants":["LG-Sleep hits 85% accuracy on mouse sleep scoring","Subject-independent EEG net nails mouse sleep stages","Local-global temporal model beats CNN, LSTM on sleep","Autoencoder trick makes sleep scoring robust with few labels","Mice sleep scoring: LG-Sleep achieves 85% accuracy and 0.75 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000326,"raw_usage":{"total_tokens":1849,"prompt_tokens":991,"completion_tokens":858,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":770}},"tokens_in":607,"tokens_out":858,"duration_ms":8090,"temperature":1.0,"reasoning_tokens":770,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:26:21.677943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 32-dimensional latent vectors of held-out mice and color them by mouse identity: if a human or a simple classifier can separate subjects more cleanly than wake/NREM/REM stages, then the autoencoder is capturing subject-specific signal, and the claimed subject-independent generalization would not hold.","supporting_citations":[{"cited_title":"Memory-enhancing properties of sleep depend on the oscillatory amplitude of norepinephrine,","cited_arxiv_id":null,"evidence_quote":"Supplies the EEG dataset, surgery details, and the definitions of the three vigilance states used in the experiments."},{"cited_title":"Automated scoring of pre-REM sleep in mice with deep learning,","cited_arxiv_id":null,"evidence_quote":"A CNN-based automated mouse sleep scorer that is re-implemented as a baseline in Table 5 and must be outperformed."},{"cited_title":"MC-SleepNet: large-scale sleep stage scoring in mice by deep neural networks,","cited_arxiv_id":null,"evidence_quote":"A CNN+LSTM mouse sleep scorer (MC-SleepNet) that is re-implemented as a baseline and represents the prior combined architecture."},{"cited_title":"Employing a long-short-term memory neural network to improve automatic sleep stage classification of pharmaco-EEG profiles,","cited_arxiv_id":null,"evidence_quote":"An LSTM-based pharmaco-EEG sleep scorer that is re-implemented as a baseline and represents the LSTM-only approach."}],"review_version":1}