REVIEW 3 major objections 4 minor 14 references
Masked Autoencoder Pretraining and BiXLSTM ResNet Architecture for PET/CT Tumor Segmentation
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that self-supervised masked-autoencoder pretraining raises mean Dice from 0.543 to 0.582 for whole-body PET/CT tumor segmentation on the AutoPET Task 1 dataset.
desk verdict MAE pretraining adds roughly 0.04 Dice for this PET/CT segmentation model—plausible, but the paper's 'significant' is unsupported by any variance or significance test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the two-stage transfer pipeline. Stage 1 uses a Masked Autoencoder (MAE), which masks patches of 3D PET/CT volumes and trains the encoder to reconstruct them, forcing it to encode global and local structure without labels. Stage 2 discards the decoder, keeps the pretrained encoder, and fine-tunes it as the backbone of a segmentation network whose other components are a bidirectional XLSTM (an extended LSTM reading slices in both directions), residual (ResNet) blocks, and a standard convolutional decoder; PET and CT volumes enter as fused input channels. The pretrained weights initialize the segmentation encoder, and the paper claims this initialization drives the Di
What would settle it
Run the same three configurations (no SSL, SSL on the encoder, SSL on the bottleneck) five times each with different random seeds, using the same 5-fold splits, and record the full Dice distribution. If the 0.039-point gap falls within one standard deviation of the two distributions, the claimed pretraining benefit is not distinguishable from training noise.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that self-supervised masked-autoencoder pretraining materially improves a PET/CT lesion segmentation model. Starting from the same BiXLSTM–ResNet UNet architecture, training from scratch reaches 0.543 mean Dice, while first pretraining the encoder with a masked autoencoder on unlabeled PET/CT and longitudinal CT scans and then fine-tuning with annotations reaches 0.582, along with lower false-negative volume and a small reduction in false-positive volume. The authors interpret this as evidence that the MAE phase learns robust modality-specific representations—anatomical structure from CT and metabolic activity from PET—that transfer to supervise
Load-bearing premise
The claimed benefit of self-supervised pretraining rests on a single comparison of mean Dice scores, 0.582 versus 0.543, with no reported variance across folds or training runs; if the two score distributions overlap, the gain could be noise.
Editorial extensions
If this is right
- If the gain is real, self-supervised pretraining on unlabeled PET/CT and longitudinal CT scans can raise Dice by about four points without any additional manual annotation.
- The reduced false-negative volume implies the pretrained model recovers more true lesions, which is the clinically more costly error in screening and staging.
- The two SSL variants differ by only 0.002 Dice, suggesting that the main benefit comes from having done pretraining at all, not from exactly where the pretrained weights are attached.
- The approach fits inside a standard self-configuring preprocessing and training pipeline, so it can be reproduced with moderate compute on a single GPU.
- Because PET and CT are treated as complementary input channels, the framework is a concrete example of multimodal fusion benefiting a supervised downstream task.
Reading between the lines
- The paper does not isolate whether the pretraining benefit comes from the PET/CT data, the longitudinal CT data, or both; an ablation pretraining on each source separately would localize the source of the gain.
- The architecture is not inherently PET/CT-specific, so the same two-stage recipe is a natural starting point for other multimodal volumetric segmentation tasks, though the paper does not test transfer.
- The 0.002 Dice gap between the encoder-attached and bottleneck-attached SSL variants suggests a follow-up question: where in the network do pretrained features matter most for lesion recovery?
- Because Task 2 longitudinal CT data enters only through pretraining, this design hints that a related but differently annotated dataset can contribute to segmentation accuracy without extra labeling effort.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-stage PET/CT lesion segmentation framework for the fourth AutoPET Challenge. In Stage 1, a Masked Autoencoder (MAE) is pretrained on unlabeled PET/CT and longitudinal CT data. In Stage 2, the pretrained encoder is integrated into a BiXLSTM–ResNet architecture with a convolutional decoder and fine-tuned on annotated AutoPET Task 1 data using a Dice-plus-cross-entropy loss, with PET and CT fused as input channels. On the AutoPET Task 1 benchmark, the authors report a mean Dice score of 0.582 with SSL pretraining versus 0.543 without, together with false-negative and false-positive volumes, and conclude that self-supervised pretraining significantly improves segmentation accuracy. The paper includes qualitative visualizations and states that code will be made available.
Significance. If the claimed improvement is reproducible, the paper would provide a useful evidence point for self-supervised pretraining in multimodal PET/CT segmentation, a clinically relevant task. The work is grounded in a public benchmark (AutoPET), uses the standard nnUNet v2 framework, and reports a two-stage architecture that is clearly described at a high level. However, the central quantitative claim rests entirely on a single comparison of mean Dice scores with no measure of variance, no per-fold results, and no significance test. Given the small absolute differences involved, the main conclusion is currently not statistically supported. The reader's stress-test concern is well-founded and is the primary obstacle to acceptance.
major comments (3)
- [Section 3, Table 2] The central claim that SSL 'significantly enhances' segmentation (Abstract; Section 3; Section 4) is supported only by mean Dice values: 0.543 (no SSL), 0.580 (SSL Enc), 0.582 (SSL Bot). No standard deviations, per-fold numbers, or significance tests are reported. The two SSL variants differ by 0.002 Dice, which suggests that small architectural or training-stochasticity differences can produce variation on this scale; the reported SSL gain of 0.039 is only about twenty times that internal spread. Without error bars, a paired test, or repeated seeds, the 0.039 gain cannot be distinguished from run-to-run noise. Please report per-fold Dice, standard deviations or confidence intervals, and a paired significance test (e.g., Wilcoxon signed-rank over folds or a bootstrap) comparing SSL versus no-SSL.
- [Section 3, Table 2] The narrative in Section 3 selects the BiXLSTM–ResNet_Bot + SSL row (Dice 0.582) for the headline result, while the other SSL row (BiXLSTM–ResNet_Enc + SSL) achieves a lower Dice (0.580) but a better false-negative volume (13.78 vs 15.08 cm³). The choice of the Bot row as the 'winner' is not justified by a pre-specified selection rule, and reporting one row while downplaying the other introduces possible selection bias. Both SSL variants should be reported transparently in the abstract and discussion, or the selection criterion should be stated.
- [Section 2.4, Table 1] The Stage 1 MAE pretraining recipe is not specified: mask ratio, patch size, number of pretraining epochs, optimizer, learning rate, and the exact mix of unlabeled PET/CT and longitudinal CT data are all absent. Since the paper's contribution is the two-stage SSL pipeline, the pretraining configuration is load-bearing for reproducibility and for attributing the Dice gain to SSL rather than to a particular pretraining hyperparameter choice. Please provide these details or cite a specific configuration.
minor comments (4)
- [References] References [9] and [10] are identical (Qayyum et al., 'Transforming Heart Chamber Imaging...'). Please remove the duplicate.
- [Figures 2 and 3] The qualitative figures show only SSL-pretrained models; they do not include a from-scratch baseline. The claim that 'these figures confirm that self-supervised pretraining improves...' is therefore not supported by the figures themselves.
- [Table 1 / Section 2.3] Some labels are inconsistent: 'BiXLSTM–ResNet_Bot + SSL' is used in Table 2 but the architecture description in Section 2.4 does not explain what 'Bot' refers to. Please define this variant.
- [Section 2.3] The paper says 'nnUNet-compatible NIfTI formatting, including training/validation splits from splits_final.json' but does not state how many folds were used for the final test evaluation; '5-fold cross-validation' is mentioned later, but the mapping between folds and the reported metrics should be explicit.
Circularity Check
No significant circularity: the central SSL-versus-scratch comparison is an internal empirical ablation measured against the external AutoPET benchmark; self-citations are programmatic, not load-bearing.
full rationale
The paper's central claim is that MAE self-supervised pretraining improves PET/CT lesion segmentation (Dice 0.582 vs 0.543, Table 2). This is an empirical ablation: the same BiXLSTM–ResNet architecture is trained either from scratch or after MAE pretraining, and both are evaluated on the same AutoPET Task 1 benchmark. No equation or construction makes the SSL result equal to its input. The MAE method is introduced via citation to external literature ([13] AMAES, and the general MAE line) as well as the authors' own prior work ([3], [9]); however, these self-citations are not load-bearing in the sense of providing a uniqueness theorem or an ansatz that by itself forces the reported Dice values. The from-scratch baseline is not derived from the SSL model, nor is the reported improvement a renamed fitting parameter. The main weakness is statistical: the paper calls the improvement 'significant' without reporting standard deviations or significance tests, and the two SSL variants differ by only 0.002 Dice. That is a correctness/evidence concern, not a circularity concern, because the comparison remains an externally anchored, falsifiable experiment. There is no self-definitional step, no fitted input renamed as a prediction, and no imported uniqueness claim. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (4)
- SSL initialization point =
encoder versus bottleneck variants
- MAE pretraining recipe (mask ratio, patch size, epochs, data mix) =
not reported
- Loss weighting for Dice plus cross-entropy =
not reported
- Patch size =
128x192x160
assumptions (4)
- domain assumption MAE self-supervised representations transfer to the downstream lesion segmentation task
- domain assumption The AutoPET Task 1 metric set (mean Dice, FN volume, FP volume) is a stable and sufficient evaluation of segmentation quality
- domain assumption Unlabeled longitudinal CT scans from Task 2 improve Task 1 generalization
- domain assumption nnUNet v2 preprocessing and augmentation defaults are appropriate for whole-body PET/CT
Cite this review
Pith. "Pith review of Masked Autoencoder Pretraining and BiXLSTM ResNet Architecture for PET/CT Tumor Segmentation." pith.science (2026). https://pith.science/paper/Z4TPHJDW
@misc{pith2026250902602,
author = {Pith},
title = {Pith review of: Masked Autoencoder Pretraining and BiXLSTM ResNet Architecture for PET/CT Tumor Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z4TPHJDW}},
note = {Machine review of arXiv:2509.02602}
}
read the original abstract
The accurate segmentation of lesions in whole-body PET/CT imaging is es-sential for tumor characterization, treatment planning, and response assess-ment, yet current manual workflows are labor-intensive and prone to inter-observer variability. Automated deep learning methods have shown promise but often remain limited by modality specificity, isolated time points, or in-sufficient integration of expert knowledge. To address these challenges, we present a two-stage lesion segmentation framework developed for the fourth AutoPET Challenge. In the first stage, a Masked Autoencoder (MAE) is em-ployed for self-supervised pretraining on unlabeled PET/CT and longitudinal CT scans, enabling the extraction of robust modality-specific representations without manual annotations. In the second stage, the pretrained encoder is fine-tuned with a bidirectional XLSTM architecture augmented with ResNet blocks and a convolutional decoder. By jointly leveraging anatomical (CT) and functional (PET) information as complementary input channels, the model achieves improved temporal and spatial feature integration. Evalua-tion on the AutoPET Task 1 dataset demonstrates that self-supervised pre-training significantly enhances segmentation accuracy, achieving a Dice score of 0.582 compared to 0.543 without pretraining. These findings high-light the potential of combining self-supervised learning with multimodal fu-sion for robust and generalizable PET/CT lesion segmentation. Code will be available at https://github.com/RespectKnowledge/AutoPet_2025_BxLSTM_UNET_Segmentation
Figures
Reference graph
Works this paper leans on
-
[1]
‘Automated Lesion Segmentation in Whole-body PET/CT and Longitudinal (autopet/ct IV)’
Küstner, Thomas, Sergios Gatidis, Ornela Megne, Michael Ingrisch, Matthias Fab- ritius, Jakob Dexl, Katharina Jeblick, et al. ‘Automated Lesion Segmentation in Whole-body PET/CT and Longitudinal (autopet/ct IV)’. Zenodo, 18 March 2025. https://doi.org/10.5281/zenodo.15045096
-
[2]
Gatidis, Sergios, Marcel Früh, Matthias P. Fabritius, Sijing Gu, Konstantin Niko- laou, Christian La Fougère, Jin Ye et al. "Results from the autoPET challenge on fully automated lesion segmentation in oncologic PET/CT imaging." Nature Ma- chine Intelligence 6, no. 11 (2024): 1396-1405
work page 2024
-
[3]
Mazher, M., Razzak, I., Qayyum, A., Tanveer, M., Beier, S., Khan, T., & Niederer, S. A. (2024). Self-supervised spatial–temporal transformer fusion based federated framework for 4D cardiovascular image segmentation. Information Fusion, 106, 102256
work page 2024
-
[4]
Extreme Cardiac MRI Analysis under Respiratory Motion: Results of the CMRxMotion Challenge
Wang, Kang, Chen Qin, Zhang Shi, Haoran Wang, Xiwen Zhang, Chen Chen, Cheng Ouyang et al. "Extreme Cardiac MRI Analysis under Respiratory Motion: Results of the CMRxMotion Challenge." arXiv preprint arXiv:2507.19165 (2025)
-
[5]
Yang, Kaiyuan, Fabio Musio, Yihui Ma, Norman Juchler, Johannes C. Paetzold, Rami Al-Maskari, Luciano Höher et al. "Benchmarking the cow with the topcow challenge: Topology-aware anatomical segmentation of the circle of willis for cta and mra." ArXiv (2024): arXiv-2312
work page 2024
-
[6]
ISLES'24: Final Infarct Prediction with Multimodal Imaging and Clinical Data. Where Do We Stand?
de la Rosa, Ezequiel, Ruisheng Su, Mauricio Reyes, Roland Wiest, Evamaria O. Riedel, Florian Kofler, Kaiyuan Yang et al. "Isles’ 24: improving final infarct pre- diction in ischemic stroke using multimodal imaging and clinical data (2024)." arXiv preprint arXiv:2408.10966
work page Pith review arXiv 2024
-
[7]
Multi -center fetal brain tissue annota- tion (feta) challenge 2022 results
Payette, Kelly, Céline Steger, Roxane Licandro, Priscille De Dumast, Hongwei Bran Li, Matthew Barkovich, Liu Li et al. "Multi -center fetal brain tissue annota- tion (feta) challenge 2022 results." IEEE transactions on medical imaging (2024)
work page 2022
-
[8]
Imran, Muhammad, Jonathan R. Krebs, Vishal Balaji Sivaraman, Teng Zhang, Amarjeet Kumar, Walker R. Ueland, Michael J. Fassler et al. "Multi-class segmen- tation of aortic branches and zones in computed tomography angiography: The aortaseg24 challenge." arXiv preprint arXiv:2502.05330 (2025)
arXiv 2025
Show all 14 references
-
[10]
Transforming 10 Heart Chamber Imaging: Self -Supervised Learning for Whole Heart Reconstruc- tion and Segmentation
Qayyum, Abdul, Hao Xu, Brian P. Halliday, Cristobal Rodero, Christopher W. Lanyon, Richard D. Wilkinson, and Steven Alexander Niederer. "Transforming 10 Heart Chamber Imaging: Self -Supervised Learning for Whole Heart Reconstruc- tion and Segmentation." arXiv preprint arXiv:24...
2024 arXiv
-
[11]
Hunting imaging biomarkers in pulmonary fibrosis: benchmarks of the AIIB23 challenge
Nan, Yang, Xiaodan Xing, Shiyi Wang, Zeyu Tang, Federico N. Felder, Sheng Zhang, Roberta Eufrasia Ledda et al. "Hunting imaging biomarkers in pulmonary fibrosis: benchmarks of the AIIB23 challenge." Medical Image Analysis 97 (2024): 103253
2024
-
[12]
A robust ensemble algorithm for ischemic stroke lesion segmentation: Generalizability and clinical utility beyond the isles challenge
de la Rosa, Ezequiel, Mauricio Reyes, Sook -Lei Liew, Alexandre Hutton, Roland Wiest, Johannes Kaesmacher, Uta Hanning et al. "A robust ensemble algorithm for ischemic stroke lesion segmentation: Generalizability and clinical utility beyond the isles challenge." arXiv preprint...
2024 arXiv
-
[13]
Amaes: Augmented masked autoencoder pretraining on public brain mri data for 3d-native segmentation
Munk, Asbjørn, Jakob Ambsdorf, Sebastian Llambias, and Mads Nielsen. "Amaes: Augmented masked autoencoder pretraining on public brain mri data for 3d-native segmentation." arXiv preprint arXiv:2408.00640 (2024)
2024 arXiv
-
[14]
nnU-Net: a self-configuring method for deep learning-based biomed- ical image segmentation
Isensee, Fabian, Paul F. Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H. Maier-Hein. "nnU-Net: a self-configuring method for deep learning-based biomed- ical image segmentation." Nature methods 18, no. 2 (2021): 203-211
2021
-
[15]
xlstm: Extended long short-term memory
Beck, Maximilian, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksan- dra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. "xlstm: Extended long short-term memory." Advances in Neural Information Processing Systems 37 (2024): 107547-107603
2024
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.