REVIEW 4 major objections 7 minor 32 references
Is Self-Pretraining really useful to improve diagnosis in medical Time Series?
T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper sets out to show that self-pretraining a transformer on the same medical time-series data it will later classify reliably beats training from scratch, with the advantage growing in deeper models and depending on how the masking…
desk verdict Empirically useful but under-specified: the SPT gains are plausible, yet the 10-epoch vs 115-epoch training budgets make the core comparison hard to interpret. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is masked-reconstruction pre-training applied in-domain: before supervised fine-tuning, the transformer is trained with a Masked Mean Squared Error loss over masked input elements only, under four masking policies—point-wise, block, column, and their union, called mixed. This procedure is what the paper calls Self-PreTraining (SPT). The four masks impose deliberately different inductive biases: local denoising, long-range temporal reasoning, cross-channel inference, and all three combined. The depth axis completes the mechanism: the paper argues SPT aligns attention toward meaningful temporal and cross-modal patterns, lowering gradient variance and loss-landscape sharpness, which is why deeper models show larger gains.
What would settle it
Re-run the univariate and multivariate pipelines with sensor selection fixed a priori or restricted strictly to training folds, and check whether the reported SPT advantages (for example, 9.9 percentage points on the Camargo gyroscope-Y signal at depth 3, or 6.8 points on the Parkinson left-foot signal at depth 2) persist; if they shrink to noise, the claim that SPT helps in the simplest single-sensor setting is not supported.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that Self-PreTraining (SPT)—pre-training a transformer to reconstruct masked elements of the same dataset it will later be fine-tuned on—is a generally useful initialization for medical time-series classification. Across three tasks (CAMARGO locomotion, Non-EEG stress, and Gait Parkinson's Disease), every SPT variant matches or beats training from scratch at depths of one to three layers, with multivariate accuracy improvements of roughly 0 to 6 percentage points and single-sensor gains as large as 9.9 points (Camargo gyroscope Y at depth 3). The benefit scales with model depth, and the best masking strategy tracks data structure: block and column masking for strongly periodic multi-sensor signals, mixed masking for heterogeneous physiological signals. The paper interprets SPT as a structured warm start that steers optimization toward useful temporal and cross-channel representations.
Load-bearing premise
The load-bearing premise is that the single 'most informative' sensor for each univariate experiment was chosen without ever looking at held-out subjects or test windows, but the paper says only that selection used domain knowledge and a preliminary performance screening.
Editorial extensions
If this is right
- SPT can be added to an existing medical time-series transformer pipeline with no architectural changes and no external data, only an extra in-domain reconstruction phase.
- Because the reported gains grow with depth, practitioners building deeper transformers for clinical data should expect SPT to matter more rather than less.
- The best masking choice depends on signal structure: block and column masks suit periodic, synchronized multi-sensor data, while mixed masks suit heterogeneous physiological signals.
- Across all tested configurations SPT never degrades performance relative to training from scratch, making it a low-risk initialization in data-limited settings.
- Even with a single sensor channel, SPT improves accuracy, so the strategy applies when only one modality is available.
Reading between the lines
- Not tested in the paper, but if SPT's benefit is mainly a warm-start effect, a cheaper comparison would pit SPT against several random restarts or extended fine-tuning from scratch to isolate what the reconstruction objective itself adds.
- The paper does not report cross-dataset transfer, but a natural test is to pre-train on one medical time-series dataset and fine-tune on another; that would separate gains due to any masked reconstruction from gains due specifically to same-data pre-training.
- If SPT acts as a partial substitute for model capacity, as the depth results suggest, then a pretrained two-layer model that matches a three-layer from-scratch model could lower deployment cost on resource-constrained clinical devices.
- The paper's optimization-based interpretation implies that other conditioning techniques, such as learning-rate warmup, weight averaging, or curriculum schedules, might reproduce part of the SPT gain, which would change the practical cost-benefit calculation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper asks whether masked-reconstruction Self-PreTraining (SPT) on the target dataset improves Transformer classification accuracy on medical time series relative to training from scratch. The authors evaluate three public datasets (Camargo locomotion, Non-EEG Stress, Gait in Parkinson's Disease), four masking strategies, and model depths of 1-3 layers. They report that SPT rarely hurts and often helps, with gains that increase with depth, and they propose an optimization-based warm-start interpretation. The main quantitative evidence is in Table III (univariate) and Table IV (multivariate), with statistical significance computed by Wilcoxon signed-rank tests.
Significance. If the central claim is established, the paper would provide a practical, data-efficient initialization strategy for clinical time-series transformers, needing no external data or architectural changes. The study's breadth is a strength: three heterogeneous datasets, four masking policies, and systematic depth variation, all on public data. The empirical tables, however, currently do not isolate the SPT objective from increased training budget, and several internal inconsistencies prevent accepting the headline results as stated. The paper also ships no code or pre-trained models, so reproducibility is limited to the tables themselves.
major comments (4)
- [IV-c, Tables III and IV] The reported accuracy gains do not isolate the SPT objective from training duration. From-scratch models are trained for 10 epochs, while SPT models receive 100 pre-training epochs plus 15 fine-tuning epochs. The early-stopping rule ('terminated if no improvement was observed for 100 consecutive epochs') cannot trigger in 10- or 15-epoch runs, so it does not establish that from-scratch models converged. The improvements in Table III (e.g., Camargo Gyr y, 3 layers: 0.524 to 0.623, a 9.9 pp gain) and Table IV (e.g., Parkinson, depth 2, SPT-Column: 0.895 to 0.974, a 7.9 pp gain) could therefore be explained by additional gradient steps rather than by the self-pretraining objective. A compute-matched or convergence-matched baseline (e.g., from-scratch training for 115 epochs, or early-stopping both regimes under the same patience) is required before the central claim can be evaluated.
- [Abstract, Tables III and IV, V-A-c] The abstract's statement that 'SPT consistently improves classification accuracy by 0-6 percentage points' is contradicted by the paper's own tables. Table III reports a 9.9 pp improvement for Camargo Gyr y at 3 layers, and Table IV shows a 7.9 pp improvement for Parkinson at depth 2 with SPT-Column. Section V-A-c also states that Parkinson LeftFoot gains 'exceed 6%' (6.8 pp). The abstract and main text need to be aligned to the actual range of reported improvements, or the range must be qualified.
- [V-A-a and V-A-b] The univariate sensor selection protocol is not fully specified. The text states that sensors were selected based on 'prior domain knowledge and preliminary performance screening,' but it does not state whether this screening used the held-out subjects or test windows. If the held-out test data influenced sensor choice, the univariate SPT gains (including the 9.9 pp Camargo Gyr-y improvement) are biased upward by test-set selection. The authors should describe the screening procedure explicitly, including which data splits were used, or use a nested validation-based selection so that the univariate claims are unbiased.
- [V-B-b and Table IV] SPT-Mixed results are quoted in the text but missing from Table IV. Section V-B-b states that 'At depth 3, accuracy reaches 0.788 (p≤0.01)' for Stress, and that SPT-Mixed 'provides the most stable improvements across heterogeneous datasets,' yet Table IV contains columns only for From Scratch, SPT-Point, SPT-Block, and SPT-Column. The missing column prevents the reader from verifying a claimed central result, namely that the mixed strategy is best for heterogeneous Stress data. Please add the SPT-Mixed results to the table or to a supplementary table, and correct any textual references accordingly.
minor comments (7)
- [III-B] The notation for the mixed masking policy uses both 'column' and 'feature' masking (e.g., M_mix = M_point(p1) ∨ M_column(p2) ∨ M_feature(p3)), while Section III-B also refers to 'column-wise masking' and 'feature masking' for the same strategy. Please define these terms consistently.
- [IV-c] The sentence 'A learning-rate sweep over of 0.01 was.' appears truncated and ungrammatical. Please state the actual learning-rate values swept and which values were selected for each regime.
- [Table III] In the Stress HR rows, the Recall column contains '0.7231' and the F1 column contains '0.7280', which appear to have an extra decimal digit compared with the other entries. Please standardize the number of decimal places.
- [II-B] The phrase '40% Stress’s' should be '40% stress.' Minor grammatical issue in the class-distribution description.
- [V-A-a] The abbreviation SPT is defined as 'Self-PreTraining' in the abstract and introduction, but Section V-A-a uses 'Structured Pre-Training.' Please use a single expansion consistently.
- [IV-d] The statistical section says the Wilcoxon signed-rank test was used, but it does not specify the pairing unit (subjects for LOSO, repeated splits for Parkinson) or the number of comparisons included in the Bonferroni correction. Please state these details so the reported p-values can be reproduced.
- [V-C] The claim that SPT 'reduces loss landscape sharpness' is presented without any direct measurement or citation. If this is an interpretation rather than a measured result, please mark it as such.
Circularity Check
No circularity found: the SPT-vs-from-scratch accuracy comparisons are measured test-set deltas, not derived quantities; the sole self-citation [2] is motivational and not load-bearing.
full rationale
The central claim is an empirical comparison (Sec. V, Tables III and IV) between transformer models initialized via masked-reconstruction SPT and models trained from scratch with Xavier initialization, on the same backbone and the same held-out evaluation protocol. The reported 0-6 percentage-point gains (and their depth scaling) are measured accuracy differences, not outputs of any fitted equation, so no prediction reduces to its own input by construction. The one self-citation, Ref. [2] (Coser et al.), appears in Sec. I-A ('In contrast, SPT denotes a setting in which the model is pre-trained directly on the same dataset that will later be used for fine tuning [1], [2]') and in the abstract's motivational sentence; it supplies the definition and framing of SPT but does not generate any accuracy number, and it is co-cited with the independent Ref. [1] (Amos et al.), which also motivates the architecture choice in Sec. III ('We choose this architecture because is the one used in the paper [1]...'). No uniqueness theorem or ansatz is imported from the authors' prior work, and no external benchmark is needed to confirm the comparison. Separate correctness risks exist but are not circularity and therefore do not raise the score: (i) Sec. V-A-a selects univariate sensors via 'prior domain knowledge and preliminary performance screening' without stating whether held-out subjects were used, so univariate gains (e.g., 9.9 pp Camargo Gyr-Y at depth 3, Table III) may be selection-biased; (ii) Sec. IV-A-c trains from-scratch for 10 epochs versus 100 pretraining plus 15 fine-tuning epochs for SPT, a compute-budget confound; (iii) Sec. V-B-b claims 'SPT-Mixed: ... At depth 3, accuracy reaches 0.788 (p≤0.01)' but Table IV contains no Mixed column, leaving that headline claim unsupported in the tables; (iv) Sec. V-B-a says multivariate models use 'all available sensor channels' while the parentheticals list only subsets; and (v) Sec. V-C asserts SPT 'reduces loss landscape sharpness' without reporting any sharpness measurement. These are experimental or reporting limitations, not self-referential derivations, so the honest circularity verdict is essentially clean with only a minor non-load-bearing self-citation.
Assumptions & free parameters
free parameters (5)
- Masking ratio p (global and per-strategy p1,p2,p3) =
Not reported
- Block length L =
Not reported
- Model dimension d and attention head count =
Not reported
- Learning-rate sweep range =
Not reported
- Number of repeated subject-wise splits (Parkinson) =
Not reported
assumptions (4)
- domain assumption The three public datasets (Camargo, Non-EEG Stress, Gait PD) are representative of medical time-series classification tasks.
- domain assumption Masked reconstruction pretraining on the same small dataset provides a meaningful signal that transfers to classification.
- domain assumption The Wilcoxon signed-rank test with Bonferroni correction over LOSO folds is a valid significance test for these comparisons.
- domain assumption Accuracy is the primary metric and macro-averaged metrics are appropriate for the multiclass Camargo task.
Cite this review
Pith. "Pith review of Is Self-Pretraining really useful to improve diagnosis in medical Time Series?." pith.science (2026). https://pith.science/paper/SW3QU3U2
@misc{pith2026260806122,
author = {Pith},
title = {Pith review of: Is Self-Pretraining really useful to improve diagnosis in medical Time Series?},
year = {2026},
howpublished = {\url{https://pith.science/paper/SW3QU3U2}},
note = {Machine review of arXiv:2608.06122}
}
read the original abstract
Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether similar gains extend to multimodal, multivariate, and even simple univariate medical time series. Our objective is to assess the impact of SPT on the performance and scalability of transformer-based models across diverse medical applications, particularly under limited data conditions. We evaluate transformer architectures on three representative medical time-series tasks: rehabilitation robotics (Camargo dataset), stress detection (Non-EEG Stress), and Parkinson's disease detection (Gait Parkinson's Disease). Models are trained either from scratch or through SPT using four masking-based objectives designed to promote temporal and cross-modal representation learning, and we systematically vary model depth to examine how capacity interacts with pre-training benefits. Across datasets and configurations, SPT consistently improves classification accuracy by 0-6 percentage points depending on masking strategy, dataset and architecture, with gains observed not only in multivariate settings but also when models are restricted to simple univariate inputs. The improvements increase for deeper models that can better exploit the enriched temporal representations learned during pre-training. These findings indicate that SPT is a simple and general strategy that enhances transformer performance on medical time-series tasks without requiring task-specific architectural changes, supporting its potential to improve robustness and accuracy in data-limited clinical settings.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Never train from scratch: Fair comparison of long-sequence models requires data-driven priors,
I. Amos, J. Berant, and A. Gupta, “Never train from scratch: Fair comparison of long-sequence models requires data-driven priors,”arXiv preprint arXiv:2310.02980, 2023
arXiv 2023
-
[2]
Towards Understanding Self-Pretraining for Sequence Classification
O. Coser, L. Zollo, P. Soda, and A. Orvieto, “Towards understanding self-pretraining for sequence classification,”arXiv preprint arXiv:2605.21070, 2026
work page Pith review arXiv 2026
-
[3]
Self-supervised pre-training for time series classification,
P. Shi, W. Ye, and Z. Qin, “Self-supervised pre-training for time series classification,” in2021 International Joint Conference on Neural Networks (IJCNN). IEEE, 2021, pp. 1–8
work page 2021
-
[4]
Timesiam: A pre-training framework for siamese time-series modeling,
J. Dong, H. Wu, Y . Wang, Y . Qiu, L. Zhang, J. Wang, and M. Long, “Timesiam: A pre-training framework for siamese time-series modeling,”arXiv preprint arXiv:2402.02475, 2024
arXiv 2024
-
[5]
Self-supervised contrastive pre-training for time series via time-frequency consistency,
X. Zhang, Z. Zhao, T. Tsiligkaridis, and M. Zitnik, “Self-supervised contrastive pre-training for time series via time-frequency consistency,” Advances in neural information processing systems, vol. 35, pp. 3988–4003, 2022
work page 2022
-
[6]
Self-supervised learning for time series analysis: Taxonomy, progress, and prospects,
K. Zhang, Q. Wen, C. Zhang, R. Cai, M. Jin, Y . Liu, J. Y . Zhang, Y . Liang, G. Pang, D. Songet al., “Self-supervised learning for time series analysis: Taxonomy, progress, and prospects,”IEEE transactions on pattern analysis and machine intelligence, vol. 46, no. 10, pp. 6775–6794, 2024
work page 2024
-
[7]
Self-supervised contrastive learning for medical time series: A systematic review,
Z. Liu, A. Alavi, M. Li, and X. Zhang, “Self-supervised contrastive learning for medical time series: A systematic review,”Sensors, vol. 23, no. 9, p. 4221, 2023
work page 2023
-
[8]
A survey on time-series pre-trained models,
Q. Ma, Z. Liu, Z. Zheng, Z. Huang, S. Zhu, Z. Yu, and J. T. Kwok, “A survey on time-series pre-trained models,”IEEE Transactions on Knowledge and Data Engineering, 2024
2024
Show all 32 references
-
[9]
Multimodal pretraining of medical time series and notes,
R. King, T. Yang, and B. J. Mortazavi, “Multimodal pretraining of medical time series and notes,” inMachine Learning for Health (ML4H). PMLR, 2023, pp. 244–255
2023
-
[10]
Contrastive pre-training for multimodal medical time series,
A. Raghu, P. Chandak, R. Alam, J. Guttag, and C. Stultz, “Contrastive pre-training for multimodal medical time series,” inNeurIPS 2022 Workshop on Learning from Time Series for Health, 2022
2022
-
[11]
Transehr: self-supervised transformer for clinical time series data,
Y . Xu, S. Xu, M. Ramprassad, A. Tumanov, and C. Zhang, “Transehr: self-supervised transformer for clinical time series data,” inMachine Learning for Health (ML4H). PMLR, 2023, pp. 623–635
2023
-
[12]
Primenet: Pre-training for irregular multivariate time series,
R. R. Chowdhury, J. Li, X. Zhang, D. Hong, R. K. Gupta, and J. Shang, “Primenet: Pre-training for irregular multivariate time series,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 6, 2023, pp. 7184–7192
2023
-
[13]
Timenet: Pre-trained deep recurrent neural network for time series classification,
P. Malhotra, V . TV , L. Vig, P. Agarwal, and G. Shroff, “Timenet: Pre-trained deep recurrent neural network for time series classification,” arXiv preprint arXiv:1706.08838, 2017
2017 arXiv
-
[14]
Large pre-trained time series models for cross-domain time series analysis tasks,
H. Prabhakar Kamarthi and B. A. Prakash, “Large pre-trained time series models for cross-domain time series analysis tasks,”Advances in Neural Information Processing Systems, vol. 37, pp. 56 190–56 214, 2024
2024
-
[15]
Simmtm: A simple pre-training framework for masked time-series modeling,
J. Dong, H. Wu, H. Zhang, L. Zhang, J. Wang, and M. Long, “Simmtm: A simple pre-training framework for masked time-series modeling,” Advances in Neural Information Processing Systems, vol. 36, pp. 29 996–30 025, 2023
2023
-
[16]
Unimts: Unified pre-training for motion time series,
X. Zhang, D. Teng, R. R. Chowdhury, S. Li, D. Hong, R. Gupta, and J. Shang, “Unimts: Unified pre-training for motion time series,” Advances in Neural Information Processing Systems, vol. 37, pp. 107 469–107 493, 2024
2024
-
[17]
Self pre-training with masked autoencoders for medical image classification and segmentation,
L. Zhou, H. Liu, J. Bae, J. He, D. Samaras, and P. Prasanna, “Self pre-training with masked autoencoders for medical image classification and segmentation,” in2023 IEEE 20th international symposium on biomedical imaging (ISBI). IEEE, 2023, pp. 1–6
2023
-
[18]
Robust lane detection through self pre-training with masked sequential autoencoders and fine-tuning with customized polyloss,
R. Li and Y . Dong, “Robust lane detection through self pre-training with masked sequential autoencoders and fine-tuning with customized polyloss,”IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 12, pp. 14 121–14 132, 2023
2023
-
[19]
Self pre-training with single-scale adapter for left atrial segmentation,
C. Tu, Z. Huang, Z. Deng, Y . Yang, C. Ma, J. He, J. Ye, H. Wang, and X. Ding, “Self pre-training with single-scale adapter for left atrial segmentation,” inChallenge on Left Atrial and Scar Quantification and Segmentation. Springer, 2022, pp. 24–35
2022
-
[20]
Self pre-training with adaptive mask autoencoders for variable-contrast 3d medical imaging,
B. K. Das, G. Zhao, H. Liu, T. J. Re, D. Comaniciu, E. Gibson, and A. Maier, “Self pre-training with adaptive mask autoencoders for variable-contrast 3d medical imaging,” in2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI). IEEE, 2025, pp. 1–5
2025
-
[21]
Masked autoencoder self pre-training for defect detection in microelectronics,
N. R ¨ohrich, A. Hoffmann, R. Nordsieck, E. Zarbali, and A. Javanmardi, “Masked autoencoder self pre-training for defect detection in microelectronics,”arXiv preprint arXiv:2504.10021, 2025
2025 arXiv
-
[22]
emotion2vec: Self-supervised pre-training for speech emotion representation,
Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen, “emotion2vec: Self-supervised pre-training for speech emotion representation,” arXiv preprint arXiv:2312.15185, 2023
2023 arXiv
-
[23]
Effectiveness of self-supervised pre-training for speech recognition,
A. Baevski, M. Auli, and A. Mohamed, “Effectiveness of self-supervised pre-training for speech recognition,”arXiv preprint arXiv:1911.03912, 2019. 21
1911 arXiv
-
[24]
Slip: Self-supervision meets language-image pre-training,
N. Mu, A. Kirillov, D. Wagner, and S. Xie, “Slip: Self-supervision meets language-image pre-training,” inEuropean conference on computer vision. Springer, 2022, pp. 529–544
2022
-
[25]
Rethinking pre-training and self-training,
B. Zoph, G. Ghiasi, T.-Y . Lin, Y . Cui, H. Liu, E. D. Cubuk, and Q. Le, “Rethinking pre-training and self-training,”Advances in neural information processing systems, vol. 33, pp. 3833–3845, 2020
2020
-
[26]
A survey on self-supervised pre-training for sequential transfer learning in neural networks,
H. H. Mao, “A survey on self-supervised pre-training for sequential transfer learning in neural networks,”arXiv preprint arXiv:2007.00800, 2020
2007 arXiv
-
[27]
Are large-scale datasets necessary for self-supervised pre-training?
A. El-Nouby, G. Izacard, H. Touvron, I. Laptev, H. Jegou, and E. Grave, “Are large-scale datasets necessary for self-supervised pre-training?” arXiv preprint arXiv:2112.10740, 2021
2021 arXiv
-
[28]
A comprehensive, open-source dataset of lower limb biomechanics in multiple conditions of stairs, ramps, and level-ground ambulation and transitions,
J. Camargo, A. Ramanathan, W. Flanagan, and A. Young, “A comprehensive, open-source dataset of lower limb biomechanics in multiple conditions of stairs, ramps, and level-ground ambulation and transitions,”Journal of Biomechanics, vol. 119, p. 110320, 2021
2021
-
[29]
A non-eeg biosignals dataset for assessment and visualization of neurological status,
J. Birjandtalab, D. Cogan, M. B. Pouyan, and M. Nourani, “A non-eeg biosignals dataset for assessment and visualization of neurological status,” in2016 IEEE International Workshop on Signal Processing Systems (SiPS). IEEE, 2016, pp. 110–114
2016
-
[30]
Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals,
A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley, “Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals,”circulation, vol. 10...
2000
-
[31]
A generalized wilcoxon test for comparing arbitrarily singly-censored samples,
E. A. Gehan, “A generalized wilcoxon test for comparing arbitrarily singly-censored samples,”Biometrika, vol. 52, no. 1-2, pp. 203–224, 1965
1965
-
[32]
What is the bonferroni correction?
M. A. Napierala, “What is the bonferroni correction?”Aaos Now, pp. 40–41, 2012
2012
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.