REVIEW 3 major objections 7 minor 38 references
CiTrus: Squeezing Extra Performance out of Low-data Bio-signal Transfer Learning
T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A CNN-transformer hybrid pretrained on EEG sets new low-data bio-signal benchmarks.
desk verdict Solid evaluation framework and a useful fine-tuning trick, but the headline frequency-pretrained model has an unexplained near-chance collapse on the original TFC splits. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is CiTrus, a hybrid of a 3-layer residual convolutional encoder and a PatchTST transformer. The CNN produces patch embeddings that are masked and fed to the transformer; channel independence is achieved by concatenating channels along the batch dimension, which lets the same pretrained weights transfer to datasets with different channel counts. Two self-supervised pretraining heads are used: one reconstructs the raw time series of masked patches, and the other reconstructs z-scored mel-spectrograms, forcing the model to predict frequency content that is not available from neighboring patches. For transfer, the fine-tuning data is resampled to the pretraining sampling frequency; if it becomes longer than the pretraining window, an overlapping sliding window is applied and embeddings are averaged across windows. This alignment is what makes the pretrained frequency filters and temporal relationships reusable.
What would settle it
Evaluate CiTrus(fp) on the EMG and FD-B benchmark splits using the same 30-second pretraining windows and the same resampling rule as bioFAME, with no per-model exceptions; if its accuracy falls to the level of the baselines (e.g., below ~90% on EMG@20%), the claimed significant margin is an artifact of protocol selection.
Extended reading notes
Core claim
The central claim is that CiTrus outperforms previous bio-signal transfer-learning models (TFC, bioFAME, SimMTM, PatchTST) by a significant margin on several low-data benchmarks. The best configuration, CiTrus with frequency-based pretraining (fp), achieves the highest average performance in the lowest and highest data regimes, including 97.92% accuracy on EMG@20%, 82.44% on ECG@0.5%, and 79.78% on FD-B@0.5%. The convolution-only encoder (Ci) trained from scratch already outperforms all baselines on several datasets, indicating that convolutional inductive bias and parameter efficiency carry much of the advantage. The paper also claims that resampling downstream data to match the pretraining frequency, with overlapping sliding windows when needed, improves fine-tuning for essentially all models, by up to about 60% on the PPG dataset, because it keeps the learned frequency representations aligned with the pretraining data.
Load-bearing premise
The headline comparisons assume every model is evaluated under the same transfer protocol, but the paper allows per-model choices of pretraining window length and resampling in the appendix, so the claimed significant margin could reflect those choices rather than the architecture alone.
Editorial extensions
If this is right
- Low-data bio-signal classifiers for EMG, ECG, PPG, and fault detection can be built by pretraining once on EEG, then fine-tuning with a fraction of the usual labeled data.
- Predicting the spectrogram of masked patches is a better pretraining task than raw-signal reconstruction for the lowest and highest data regimes, so future self-supervised designs should consider frequency-domain targets.
- Resampling downstream data to the pretraining sampling rate, rather than interpolating to the pretraining length, is a nearly universal improvement and should become the default transfer protocol.
- Convolution-based encoders outperform transformer-only encoders in low-data bio-signal settings, suggesting that weight sharing and parameter efficiency matter more than attention capacity when labels are scarce.
- Pretraining helps transformer-heavy models the most, especially in low-data regimes, while fully convolutional models trained from scratch already work well on some datasets (EMG, FD-B).
Reading between the lines
- The channel-batch concatenation trick could generalize to any multivariate time-series problem, making pretrained sensor models portable across devices with different channel counts.
- The success of frequency pretraining suggests a broader principle for signals where discriminative information lives in the spectrum: self-supervised targets should be spectral, not raw, which could carry over to audio and vibration monitoring.
- Because the convolution-only encoder sometimes matches the hybrid, a testable extension is to ablate the transformer depth or replace it with a lighter mixer in extreme low-data regimes to cut parameters further.
- The reported margins may depend on per-model protocol choices (2s vs 30s pretraining windows, resampling rules); a controlled re-run with a single fixed protocol per dataset would give a cleaner ranking of models.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CiTrus, a convolution-transformer hybrid for transfer learning from SleepEDF EEG to low-data bio-signal classification tasks. It introduces masked autoencoding, a frequency-based spectrogram prediction variant, multimodal EEG/EOG pre-training, and a frequency-matching resampling fine-tuning strategy. Experiments compare against PatchTST, bioFAME, NLPatchTST, SimMTM, and a CNN-only variant across EMG, ECG, PPG, HAR, FD-B, Epilepsy, Gesture, and SleepEDF, under multiple data regimes and with both 10-fold cross-validation and the original TFC splits. The paper claims a significant performance margin for CiTrus, especially for the frequency-pretrained variant, and advocates 10-fold evaluation for transfer-learning benchmarks.
Significance. If the results hold, the paper makes useful contributions: a new hybrid architecture, two pre-training objectives, a resampling-based fine-tuning method that improves on temporal interpolation, and a more thorough evaluation protocol with cross-validation and variance reporting. The statistical reporting (standard deviations, Wilcoxon tests, fold-vs-seed variance analysis) is stronger than is typical for this literature and deserves credit. However, the headline claims currently rest on unresolved internal inconsistencies and protocol asymmetries that must be fixed before the contribution can be accepted.
major comments (3)
- [Section 4, Table 1 vs Table 5, Appendix B] Table 1 reports CiTrus(fp) as the best model on EMG@20% (ACC 97.92, ROC 99.02, PRC 98.76) and FD-B@0.5% (ACC 79.78), while Table 5 reports CiTrus(fp) at ACC 54.27 / ROC 76.11 / PRC 72.11 on EMG and ACC 30.84 / ROC 53.11 / PRC 36.88 on FD-B under the original TFC split. Both evaluations use the same stated setting for CiTrus(fp) in Appendix B: 2s pre-training windows and 200-timestep interpolation, which is the setting that Appendix G identifies as preferred for this model. The two tables therefore contradict each other for the model that underpins the abstract claim and the Table 13 averages, and the manuscript offers no explanation. The authors must either reconcile these results or replace the affected claims with results from a reproducible protocol; a near-chance result on the full TFC split for a model that is near-perfect under 10-fold CV suggests an implementation issue that cannot remain unaddressed.
- [Appendices B and G] The headline comparison is not run under a single transfer protocol. For EMG and FD-B, Appendix B states that most models use 30s SleepEDF pre-training with resampling to 3000 timesteps, while SimMTM and CiTrus(fp) use 2s pre-training because of incompatibility or better performance. Appendix G then shows large protocol-dependent differences, e.g., moving from 2s to 30s pre-training improves PatchTST(s) on EMG@20% by +85.7% while changing CiTrus(fp) by -7.2% to -9.8%. If these per-model protocol choices were made after inspecting test performance, the claimed significant margin over baselines is not a controlled comparison. Please report all models under the same pre-training/fine-tuning protocol per dataset, or justify protocol choices using only validation-based model selection.
- [Section 4, Appendix A, SleepEDF evaluation] SleepEDF is used both as the pre-training corpus and as a downstream evaluation target, but the manuscript does not explicitly state whether the SleepEDF test set or the subjects in it were excluded from pre-training. If the test set or its subjects were included in the pre-training data, the SleepEDF downstream results and the dataset-averaged results in Table 13 are affected by information leakage. The authors should state the exact subject/window split used for pre-training versus downstream evaluation, and, if necessary, re-run the SleepEDF experiments with a properly held-out pre-training split.
minor comments (7)
- [Section 1, contributions] The second contribution bullet contains a typo: 'propsed' should be 'proposed'.
- [Section 4, Experimental settings] The sentence 'during we evaluate each model across a range of data percentages' is missing a word; it should read 'during fine-tuning, we evaluate' or similar.
- [Section 5, Discussion] The phrase 'middle data regie' should be 'middle data regime'.
- [Appendix F] Tables 15-17 report Wilcoxon p-values only for accuracy, while the text refers to 'average performance improvement' across three metrics; please clarify whether the tests were run on each metric separately or only on accuracy.
- [Figure 1] The figure is dense and the subfigure labels are small; increasing font size and resolution would make the architecture and pre-training pipelines substantially easier to follow.
- [General] No code or data link is provided. For an empirical benchmark paper of this type, releasing the code is important for reproducibility and would also help resolve the Table 1/Table 5 discrepancy discussed above.
- [Table 5 caption] The caption should state explicitly that the EMG/FD-B protocol in this table deviates from Table 1 for some models (as described in Appendix B), so readers do not interpret the table as a like-for-like comparison.
Circularity Check
No significant circularity: the paper's claims are empirical benchmark comparisons against external baselines, with no derivation that reduces to its own inputs.
full rationale
CiTrus is an empirical systems paper: the central claims (architecture outperforms baselines; frequency-based pre-training helps; resampling-adaptive fine-tuning improves transfer) are supported by measured accuracies/ROC/PRC on external datasets, not by a mathematical derivation whose conclusion is assumed in its premises. The pre-training objectives (masked auto-encoding, frequency-based spectrogram prediction) are defined independently of the downstream test metrics, and no parameter is fitted to a target and then renamed as a prediction. The paper cites prior work (TFC, bioFAME, SimMTM, PatchTST) as baselines and building blocks, but does not rely on any self-citation chain to justify its central claim; the authors' own prior work is not cited as the load-bearing evidence. The mild evaluation concerns noted in the manuscript—using SleepEDF both as the pre-training source and as one downstream evaluation target, choosing per-model 2s vs 30s pre-training windows based on performance (Appendix G), and the replication gap admitted in Appendix B—are validity or reproducibility issues, not circularity. Likewise, the large discrepancy between CiTrus(fp)'s Table 1 and Table 5 results is an internal-consistency concern that should be investigated, but it does not make the derivation circular. Under the rubric, no step can be quoted where an output is equivalent to an input by construction, so the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Pre-training window length (2s vs 30s) =
2s or 30s, chosen per model/dataset
- Spectrogram hyperparameters for frequency pre-training =
hop=4/8, n_fft=200/1024, n_mels=64/128, fmax=20, offset/scale 40
- Masking ratio and block masking size =
mask ratio 0.5, block masking size 5
- Fine-tuning dropout =
0.5 added before linear head
assumptions (3)
- domain assumption Pre-training on SleepEDF EEG transfers to other bio-signals because low-level frequency statistics are similar across bio-signals.
- domain assumption Channel-independent processing by concatenating channels along the batch preserves cross-dataset transferability.
- domain assumption The downstream benchmark splits and labels from TFC and other sources are correct and representative.
Cite this review
Pith. "Pith review of CiTrus: Squeezing Extra Performance out of Low-data Bio-signal Transfer Learning." pith.science (2026). https://pith.science/paper/7O4SXRME
@misc{pith2026241211695,
author = {Pith},
title = {Pith review of: CiTrus: Squeezing Extra Performance out of Low-data Bio-signal Transfer Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/7O4SXRME}},
note = {Machine review of arXiv:2412.11695}
}
read the original abstract
Transfer learning for bio-signals has recently become an important technique to improve prediction performance on downstream tasks with small bio-signal datasets. Recent works have shown that pre-training a neural network model on a large dataset (e.g. EEG) with a self-supervised task, replacing the self-supervised head with a linear classification head, and fine-tuning the model on different downstream bio-signal datasets (e.g., EMG or ECG) can dramatically improve the performance on those datasets. In this paper, we propose a new convolution-transformer hybrid model architecture with masked auto-encoding for low-data bio-signal transfer learning, introduce a frequency-based masked auto-encoding task, employ a more comprehensive evaluation framework, and evaluate how much and when (multimodal) pre-training improves fine-tuning performance. We also introduce a dramatically more performant method of aligning a downstream dataset with a different temporal length and sampling rate to the original pre-training dataset. Our findings indicate that the convolution-only part of our hybrid model can achieve state-of-the-art performance on some low-data downstream tasks. The performance is often improved even further with our full model. In the case of transformer-based models we find that pre-training especially improves performance on downstream datasets, multimodal pre-training often increases those gains further, and our frequency-based pre-training performs the best on average for the lowest and highest data regimes.
Figures
Reference graph
Works this paper leans on
-
[1]
G.; Lehnertz, K.; Mormann, F.; Rieke, C.; David, P.; and Elger, C
Andrzejak, R. G.; Lehnertz, K.; Mormann, F.; Rieke, C.; David, P.; and Elger, C. E. 2001. Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E, 64(6): 061907
2001
-
[2]
Buitinck, L.; Louppe, G.; Blondel, M.; Pedregosa, F.; Mueller, A.; Grisel, O.; Niculae, V.; Prettenhofer, P.; Gramfort, A.; Grobler, J.; Layton, R.; VanderPlas, J.; Joly, A.; Holt, B.; and Varoquaux, G. 2013. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Le...
work page 2013
-
[3]
Casselman, J.; Onopa, N.; and Khansa, L. 2017. Wearable healthcare: Lessons from the past and a peek into the future. Telematics and Informatics, 34(7): 1011--1023
work page 2017
-
[4]
H.; Celka, P.; Farukh, B.; Chowienczyk, P.; and Alastruey, J
Charlton, P. H.; Celka, P.; Farukh, B.; Chowienczyk, P.; and Alastruey, J. 2018. Assessing mental stress from the photoplethysmogram: a numerical study. Physiological measurement, 39(5): 054001
work page 2018
-
[5]
Chien, H.-Y. S.; Goh, H.; Sandino, C. M.; and Cheng, J. Y. 2022. Maeeg: Masked auto-encoder for eeg representation learning. arXiv preprint arXiv:2211.02625
arXiv 2022
-
[6]
Deldari, S.; Spathis, D.; Malekzadeh, M.; Kawsar, F.; Salim, F.; and Mathur, A. 2023. Latent Masking for Multimodal Self-supervised Learning in Health Timeseries. arXiv preprint arXiv:2307.16847
work page Pith review arXiv 2023
-
[7]
Dong, J.; Wu, H.; Zhang, H.; Zhang, L.; Wang, J.; and Long, M. 2024. Simmtm: A simple pre-training framework for masked time-series modeling. Advances in Neural Information Processing Systems, 36
2024
-
[8]
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929
arXiv 2020
Show all 38 references
-
[9]
C.; Venneri, A.; He, X.; Li, S.; Zhao, Y.; et al
Durongbhan, P.; Zhao, Y.; Chen, L.; Zis, P.; De Marco, M.; Unwin, Z. C.; Venneri, A.; He, X.; Li, S.; Zhao, Y.; et al. 2019. A dementia classification framework using frequency and time-frequency features based on EEG signals. IEEE Transactions on Neural Systems and Rehabilita...
2019
-
[10]
Eldele, E.; Ragab, M.; Chen, Z.; Wu, M.; Kwoh, C.-K.; Li, X.; and Guan, C. 2023. Self-supervised contrastive representation learning for semi-supervised time-series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence
2023
-
[11]
M.; Mackellar, G.; Ghane, S.; Irtza, S.; Nguyen, N.; and Salehi, M
Foumani, N. M.; Mackellar, G.; Ghane, S.; Irtza, S.; Nguyen, N.; and Salehi, M. 2024. Eeg2rep: enhancing self-supervised EEG representation through informative masked inputs. arXiv preprint arXiv:2402.17772
2024 arXiv
-
[12]
L.; Amaral, L
Goldberger, A. L.; Amaral, L. A.; Glass, L.; Hausdorff, J. M.; Ivanov, P. C.; Mark, R. G.; Mietus, J. E.; Moody, G. B.; Peng, C.-K.; and Stanley, H. E. 2000. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. circul...
2000
-
[13]
Gong, L.; Li, M.; Zhang, T.; and Chen, W. 2023. EEG emotion recognition using attention-based convolutional transformer neural network. Biomedical Signal Processing and Control, 84: 104835
2023
-
[14]
Guibas, J.; Mardani, M.; Li, Z.; Tao, A.; Anandkumar, A.; and Catanzaro, B. 2021. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587
2021 arXiv
-
[15]
He, K.; Chen, X.; Xie, S.; Li, Y.; Doll \'a r, P.; and Girshick, R. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 16000--16009
2022
-
[16]
H.; Tuk, B.; Kamphuisen, H
Kemp, B.; Zwinderman, A. H.; Tuk, B.; Kamphuisen, H. A.; and Oberye, J. J. 2000. Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the EEG. IEEE Transactions on Biomedical Engineering, 47(9): 1185--1194
2000
-
[17]
K.; Zimmer, D.; and Sextro, W
Lessmeier, C.; Kimotho, J. K.; Zimmer, D.; and Sextro, W. 2016. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM Society European Conference,...
2016
-
[18]
Li, Y.; Li, X.; Ratcliffe, M.; Liu, L.; Qi, Y.; and Liu, Q. 2011. A real-time EEG-based BCI system for attention recognition in ubiquitous environment. In Proceedings of 2011 international workshop on Ubiquitous affective awareness and intelligent interaction, 33--40
2011
-
[19]
Li, Z.; Kovachki, N.; Azizzadenesheli, K.; Liu, B.; Bhattacharya, K.; Stuart, A.; and Anandkumar, A. 2020. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895
2020 arXiv
-
[20]
Liu, J.; Zhong, L.; Wickramasuriya, J.; and Vasudevan, V. 2009. uWave: Accelerometer-based personalized gesture recognition and its applications. Pervasive and Mobile Computing, 5(6): 657--675
2009
-
[21]
L.; Pouransari, H.; Sandino, C.; Nie, J.; Goh, H.; Azemi, E.; and Moin, A
Liu, R.; Zippi, E. L.; Pouransari, H.; Sandino, C.; Nie, J.; Goh, H.; Azemi, E.; and Moin, A. 2023. Frequency-aware masked autoencoders for multimodal pretraining on biosignals. arXiv preprint arXiv:2309.05927
2023 arXiv
-
[22]
P.; McVicar, M.; Battenberg, E.; and Nieto, O
McFee, B.; Raffel, C.; Liang, D.; Ellis, D. P.; McVicar, M.; Battenberg, E.; and Nieto, O. 2015. librosa: Audio and music signal analysis in python. In SciPy, 18--24
2015
-
[23]
D.; Giannakeas, N.; and Tzallas, A
Miltiadous, A.; Gionanidis, E.; Tzimourta, K. D.; Giannakeas, N.; and Tzallas, A. T. 2023. DICE-net: a novel convolution-transformer architecture for Alzheimer detection in EEG signals. IEEE Access
2023
-
[24]
Moody, G. 1983. A new method for detecting atrial fibrillation using RR intervals. Proc. Comput. Cardiol., 10: 227--230
1983
-
[25]
Neyshabur, B.; Sedghi, H.; and Zhang, C. 2020. What is being transferred in transfer learning? Advances in neural information processing systems, 33: 512--523
2020
-
[26]
H.; Sinthong, P.; and Kalagnanam, J
Nie, Y.; Nguyen, N. H.; Sinthong, P.; and Kalagnanam, J. 2022. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730
2022 arXiv
-
[27]
D.; O'Sullivan, J
Odinaka, I.; Lai, P.-H.; Kaplan, A. D.; O'Sullivan, J. A.; Sirevaag, E. J.; Kristjansson, S. D.; Sheffield, A. K.; and Rohrbaugh, J. W. 2010. ECG biometrics: A robust short-time frequency analysis. In 2010 IEEE International Workshop on Information Forensics and Security, 1--6. IEEE
2010
-
[28]
Y.; Yao, Y.; and Dauwels, J
Peh, W. Y.; Yao, Y.; and Dauwels, J. 2022. Transformer convolutional neural networks for automated artifact detection in scalp EEG. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 3599--3602. IEEE
2022
-
[29]
Reyes-Ortiz, J.; Anguita, D.; Oneto, L.; and Parra, X. 2015. Smartphone-based recognition of human activities and postural transitions data set. UCI Machine Learning Repository. School Inf. Comput. Sci. Univ. California at Irvine, Irvine, CA, USA, available online: http://arch...
2015
-
[30]
Schmidt, P.; Reiss, A.; Duerichen, R.; Marberger, C.; and Van Laerhoven, K. 2018. Introducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM international conference on multimodal interaction, 400--408
2018
-
[31]
Song, Y.; Zheng, Q.; Liu, B.; and Gao, X. 2022. EEG conformer: Convolutional transformer for EEG decoding and visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31: 710--719
2022
-
[32]
Weiderpass, H.; Pachi, C.; Yamamoto, J.; Hamamoto, A.; Onodera, A.; and Sacco, I. 2013. Time-frequency analysis methods for detecting effects of diabetic neuropathy. International Journal for Numerical Methods in Biomedical Engineering, 29(9): 1000--1010
2013
-
[33]
A.; Moreno, A.; Wei, H.; Marlin, B
Xu, M. A.; Moreno, A.; Wei, H.; Marlin, B. M.; and Rehg, J. M. 2023. Retrieval-Based Reconstruction For Time-series Contrastive Learning. arXiv preprint arXiv:2311.00519
2023 arXiv
-
[34]
Yang, C.; Westover, M.; and Sun, J. 2024. Biot: Biosignal transformer for cross-data learning in the wild. Advances in Neural Information Processing Systems, 36
2024
-
[35]
N.; and Lopez-Paz, D
Zhang, H.; Cisse, M.; Dauphin, Y. N.; and Lopez-Paz, D. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412
2017 arXiv
-
[36]
Zhang, X.; Zhao, Z.; Tsiligkaridis, T.; and Zitnik, M. 2022. Self-supervised contrastive pre-training for time series via time-frequency consistency. Advances in Neural Information Processing Systems, 35: 3988--4003
2022
-
[37]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[38]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.