Pith. sign in

REVIEW 5 major objections 6 minor 18 references

Brain Hematoma Marker Recognition Using Multitask Learning: SwinTransformer and Swin-Unet

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Swin Transformer trained at once for classification, segmentation, and reconstruction recognizes brain hematoma markers better than a single-task Swin classifier.

desk verdict Honest small extension, but the headline 'outperformed' depends on switching F1 and AUC between settings, and the single-split evaluation is too thin to trust. read the letter →

arxiv 2505.06185 v2 pith:MNGUB4GZ submitted 2025-05-09 cs.LG cs.CV

Kodai Hirata , Tsuyoshi Okita This is my paper · ORCID
classification cs.LGcs.CV
keywords multitasklearningSwinTransformerSwin-Unetbrainhematomahypodensitysemanticsegmentationimagereconstructioncovariateshift
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that MTL-Swin-Unet, a Swin-Unet trained jointly to classify, segment, and reconstruct brain CT images, learns a representation that recognizes the hypodensity marker better than a plain Swin Transformer classifier. On test slices drawn from the same patients as the training data, the three-task model achieves the highest accuracy and F1 among the compared classifiers. On slices from hospitals the model never saw, it achieves the highest AUC. If this holds, auxiliary pixel-level objectives, segmentation especially, can counteract the spurious correlations that make such models look at the skull instead of the lesion.

What carries the argument

The central object is MTL-Swin-Unet: a U-shaped Swin Transformer network with a shared encoder whose representations feed three heads, a classification head on global average pooled features, a segmentation decoder, and an image reconstruction decoder. The three task losses are combined as a weighted sum, $L = \lambda_{\mathrm{cls}} L_{\mathrm{cls}} + \lambda_{\mathrm{seg}} L_{\mathrm{seg}} + \lambda_{\mathrm{rec}} L_{\mathrm{rec}}$, with the segmentation loss itself a weighted mix of cross-entropy and Dice loss. The shared encoder is initialized from ImageNet pretrained weights, and the decoder structure follows Swin-Unet with skip connections, letting dense pixel-level objectives regularize the classification representation.

What would settle it

Re-run the comparison with patient-level splits instead of slice-level splits: train on all patients from hospitals 1-4 and test on a fresh random draw of slices from hospitals 5-11, with the positive ratio matched by a documented procedure and multiple draws to estimate variance; if MTL-Swin-Unet no longer beats Swin Transformer in AUC, the reported advantage is an artifact of the chosen subset.

Watch

Extended reading notes

Core claim

The central claim is that multitask learning with segmentation and image reconstruction improves Swin-based classification of hypodensity in brain CT images. The paper reports that MTL-Swin-Unet (cls + seg + rec) reaches an accuracy of 0.961, precision 0.852, recall 0.809, and F1 of 0.903 on the same-hospital test set, ahead of every classifier it is compared with, and an AUC of 0.799 on the held-out-hospital test set, also the highest. The comparison also shows that the segmentation task is the beneficial auxiliary objective, while reconstruction alone reduces AUC relative to the single-task Swin Transformer; only when combined with segmentation does reconstruction add a small gain on unseen hospitals.

Load-bearing premise

The headline cross-hospital AUC result rests on the assumption that the 179 images selected from hospitals 5-11, matched to the positive ratio of the hospital-1 test set, are a fair and representative sample of the covariate shift; if this selection procedure biased the sample, the comparison would collapse.

Editorial extensions

If this is right

  • MTL-Swin-Unet can be deployed directly as a classifier even when segmentation masks are only available for a subset of training images, because the segmentation loss is averaged only over masked images.
  • Because the segmentation objective shifts the classifier's attention from skull and brain texture to the hematoma region, the resulting predictions may be more clinically trustworthy in settings where the training and test images come from different scanners or hospitals.
  • Adding reconstruction alone is not enough: the paper's comparison suggests that reconstruction should be paired with segmentation to help cross-hospital performance.
  • The method provides a practical middle ground between plain transfer learning and full domain adaptation for hospital-to-hospital generalization in medical imaging.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if segmentation is the main driver, this multitask recipe may transfer to other lesion types or imaging modalities wherever pixel-level annotations exist, even if only for a fraction of the data.
  • Editorial inference: the same-patient test split is at slice level, not patient level; a patient-level split would remove the strong within-patient correlation and could change the size of the reported F1 gap.
  • Editorial inference: the covariate-shift test subset was assembled by matching the positive ratio of the hospital-1 test set, but a more stable estimate of the AUC advantage would come from repeated random draws of such subsets with reported variance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript proposes two Transformer-based methods for classifying hypodensity markers in brain CT: MTL-Swin-Unet, which extends Swin-Unet by adding classification and image-reconstruction heads trained jointly with the segmentation head via the weighted loss in Eq. (5), and Joint-SwinTransformer, which concatenates representations from a frozen Swin-Unet segmentation encoder with a trainable SwinTransformer encoder for classification. The experiments use a private dataset of 11,780 CT images from 11 hospitals. On test data from hospitals 1-4, MTL-Swin-Unet (cls+seg+rec) achieves the highest reported F1 (0.903) and accuracy (0.961); on test data from hospitals 5-11, it achieves the highest reported AUC (0.799). The authors interpret these results as evidence that multitask learning with segmentation and reconstruction improves classification and mitigates spurious correlation under covariate shift.

Significance. The paper addresses a practically important question: whether auxiliary segmentation and reconstruction objectives improve a Swin-based classifier for a scarce medical label, and whether such multitask training helps under cross-hospital distribution shift. The strengths include a reasonably detailed architecture description and a head-to-head empirical comparison against standard baselines (ResNet152, Swin Transformer, SwinTransformer-tiny, joint-CNN) on the same private dataset. The reported differences, however, are small and are evaluated on a single split without error bars or significance tests. If the evaluation is made robust and the advantage persists, the result would be a useful empirical contribution to multitask learning for medical imaging. The manuscript does not provide code or a reproducibility statement, and the data are private, so independent verification is not currently possible.

major comments (5)
  1. [Result, Table 1] The abstract and conclusion claim that the proposed method outperformed other classifiers in F1 under no covariate shift and in AUC under covariate shift, but Table 1 shows that the advantage is metric-dependent: on Hospitals 1-4, MTL-Swin-Unet (cls+seg+rec) has the best F1 (0.903) but not the best AUC (0.967 vs Joint-SwinTransformer's 0.974); on Hospitals 5-11, it has the best AUC (0.799) but not the best F1 (0.618 vs Joint-SwinTransformer-tiny's 0.626). The claim therefore holds only when a different metric is selected as the headline in each setting. The authors should pre-specify one primary metric or report both metrics with a consistent decision rule, and discuss the trade-off explicitly.
  2. [Settings of Two Datasets] The experimental design does not rule out patient-level leakage in the same-hospital test setting. The text states that for hospital 1, 179 images were allocated as test data while the remaining 9/10 of hospital 1 was used for training, and the table footnote says non-overlapping slices from the same patient are allowed for both training and test data. If slices from the same patient appear in both training and test, the reported F1 advantage (0.903 vs 0.895) may be inflated by memorization of patient-specific features. The authors should either use a patient-stratified split for the no-covariate-shift setting or explicitly report the number of patients and the overlap statistics.
  3. [Settings of Two Datasets] The covariate-shift test set of 179 images from hospitals 5-11 is described only as randomly selected with a positive ratio matching the facility-1 test data. The manuscript does not state the total number of available images or patients, the selection algorithm, or whether selection was stratified by hospital and patient. Because the central cross-hospital AUC claim rests on this sample, the authors must document the selection procedure and report the class distribution, hospital distribution, and patient counts. Without this, the sample cannot be judged as a fair covariate-shift benchmark.
  4. [Verification of Prediction Basis, Figure 4] The claim that the model avoids spurious correlations (e.g., focusing on the skull) is supported only by two illustrative Grad-CAM examples. The paper provides no quantitative comparison of Grad-CAM maps against the segmentation masks, nor a comparison of heatmaps between the proposed method and a baseline classifier that reportedly focuses on the skull. The authors should add a quantitative faithfulness or overlap metric (e.g., Dice or IoU between the Grad-CAM region and the lesion mask) over the test set, or temper the spurious-correlation claim accordingly.
  5. [Loss Function, Eq. (5)] The multitask loss weights (lambda_cls = 0.3, lambda_seg = 0.4, lambda_rec = 0.4) are hand-set, and no sensitivity analysis or ablation over these weights is reported. Because the central comparison is between task combinations, the reader cannot determine whether the observed benefit of adding segmentation and reconstruction is robust to the weight choice. The authors should include a small sensitivity study over the lambda values or provide a principled rationale for the chosen values.
minor comments (6)
  1. [Abstract and throughout] There are typos and terminology inconsistencies: 'sprious-correlation' should be 'spurious correlation', and 'covariance shift' should be 'covariate shift' in the abstract, the Settings of Two Datasets section, and the Conclusion.
  2. [Impact of Model Size and Method] The text refers to 'Table ??' and to 'Section .' as unresolved cross-references; these placeholders should be fixed before a version is submitted for review.
  3. [Table 1 footnote and Settings of the Learner] The table footnote describes joint-CNN as based on EfficientNet, while the main text calls it 'CNN based joint-Learning' and contains the placeholder 'avoid reference for anonymity'; this should be resolved and the baseline described consistently.
  4. [Dataset] The dataset description says '8/10 of the 9,912 images from facilities 2 to 4' were used for training; please clarify whether the remaining 2/10 are validation or excluded, and how the stated total of 11,780 images is composed.
  5. [Settings of the Learner] The epoch counts are given as 600 for classification and 200 for segmentation, but it is not stated how many epochs are used for the joint multi-task models; please specify the training schedule and any early-stopping criterion for every method.
  6. [Figure 4] The caption for panel (b), 'hematoma excluding hypodensity', would be clearer as 'hematoma without hypodensity', and the figure should state the CT window/level used for display.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central comparison is an empirical benchmark against external baselines on held-out test data.

full rationale

This paper makes an empirical architecture-comparison claim: a Swin-Unet trained jointly for classification, segmentation, and reconstruction is compared with external baselines (ResNet152, Swin Transformer, Swin-Unet, Unet, joint-CNN, Joint-SwinTransformer) on two held-out test sets. There is no derivation of a predicted quantity from a fitted input; the only tunable quantities are loss weights, which are stated as fixed hand-set values (lambda_cls = 0.3, lambda_seg = 0.4, lambda_rec = 0.4) rather than fitted to test labels. Segmentation masks and reconstruction targets are auxiliary objectives derived from the same images and do not by construction determine the classification output. The mention of "joint-CNN (avoid reference for anonymity)" is a self-reference, but the comparison that supports the claim is against the listed external baselines and held-out slices, so that self-reference is not load-bearing. The weakness noted by the reader, that F1 is best in one setting and AUC in the other, concerns metric selection and statistical robustness, not circularity. Therefore no specific step reduces to its own inputs, and the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on hand-set loss weights, trusted clinical annotations, ImageNet initialization, and a single-split evaluation. No new physical entities or formal axioms are introduced; the main uncharged parameters are the multi-task weights and model hyperparameters.

free parameters (4)
  • Multi-task loss weights (lambda_cls, lambda_seg, lambda_rec) = 0.3/0.4/0.4 for cls+seg+rec; 0.4/0.6 for two-task variants
    Hand-chosen weights in Eq. 5. No sensitivity analysis is reported, so the ranking could depend on these values.
  • Segmentation loss weights (lambda_ce, lambda_dice) = 0.4 and 0.6
    Hand-chosen weights in Eq. 3 for the auxiliary segmentation loss.
  • Model size (Swin blocks and channels) = Default: 2 blocks and 96 channels; tiny: 6 blocks and 96 channels
    Architecture configuration taken from Swin-Unet; the paper states model size had little effect, so this is not a tuned parameter for the central claim.
  • Learning rate and batch size = 0.01 or 0.004 and batch sizes 32, 64, or 128 depending on model
    Hyperparameters vary between methods, so baseline comparisons may be confounded by tuning differences.
assumptions (4)
  • domain assumption Clinical labels and segmentation masks from the 11 institutions are correct and consistent.
    Training and evaluation rely on specialist annotations; no adjudication or inter-rater reliability is reported (Dataset section).
  • domain assumption ImageNet-pretrained Swin Transformer weights transfer to head CT images.
    All Swin-based encoders are initialized with ImageNet pretraining (Settings of the Learner).
  • domain assumption A single split into 179-image test sets is representative enough for the reported comparisons.
    All conclusions are drawn from one split with no repeated runs or confidence intervals (Result, Table 1).
  • ad hoc to paper Weighted-sum loss over tasks is a valid objective for this multi-task setup.
    Eq. 5 combines losses with fixed weights and averages segmentation loss only over images with masks; these choices are asserted without justification (Method, Loss Function).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Brain Hematoma Marker Recognition Using Multitask Learning: SwinTransformer and Swin-Unet." pith.science (2026). https://pith.science/paper/MNGUB4GZ

@misc{pith2026250506185,
  author       = {Pith},
  title        = {Pith review of: Brain Hematoma Marker Recognition Using Multitask Learning: SwinTransformer and Swin-Unet},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MNGUB4GZ}},
  note         = {Machine review of arXiv:2505.06185}
}
read the original abstract

This paper proposes a method MTL-Swin-Unet which is multi-task learning using transformers for classification and semantic segmentation. For spurious-correlation problems, this method allows us to enhance the image representation with two other image representations: representation obtained by semantic segmentation and representation obtained by image reconstruction. In our experiments, the proposed method outperformed in F-value measure than other classifiers when the test data included slices from the same patient (no covariate shift). Similarly, when the test data did not include slices from the same patient (covariate shift setting), the proposed method outperformed in AUC measure.

Figures

Figures reproduced from arXiv: 2505.06185 by the authors.

Figure 1
Figure 1. Architecture of MTL-Swin-Unet In classification tasks, we calculated the loss Lcls using the cross-entropy function, similar to the first loss function in segmentation tasks. Multi-task learning typically optimizes by taking a weighted sum of the losses for each task to find the minimum value. The final loss is computed as a weighted sum of the individual task losses, as expressed by the follow￾ing equation. L = λcl… view at source ↗
Figure 2
Figure 2. Architecture of Joint-SwinTransformer dimension of the linear layer has been modified to accom￾modate concatenated representations. Loss Function Similar to the first method, for the prelim￾inary segmentation task, a loss function combining cross￾entropy and Dice loss is employed. For the classification task, the cross-entropy function is used. Experiment In this chapter, we conducted experiments using two meth￾ods … view at source ↗
Figure 3
Figure 3. (a) Hypodensity (b) Images without hypodensity overlapping with other hemorrhage classes (from left to right: image [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The visualization results of Grad-CAM using MTL-Swin-Unet (cls + seg + rec) for (a) hypodensity images and (b) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 6 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Belharbi, S.; Ben Ayed, I.; McCaffrey, L.; and Granger, E. 2021. Deep active learning for joint classification & segmentation with weak annotator. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 3338--3347

    work page 2021

  4. [4]

    Bhattacharjee, D.; Zhang, T.; S \"u sstrunk, S.; and Salzmann, M. 2022. Mult: An end-to-end multitask learning transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12031--12041

    work page 2022

  5. [5]

    B.; Charidimou, A.; Jessel, M

    Boulouis, G.; Morotti, A.; Brouwers, H. B.; Charidimou, A.; Jessel, M. J.; Auriel, E.; Pontes-Neto, O.; Ayres, A.; Vashkevich, A.; Schwab, K. M.; et al. 2016. Association between hypodensities detected by computed tomography and hematoma expansion in patients with intracerebral hemorrhage. JAMA neurology, 73(8): 961--968

    work page 2016

  6. [6]

    Cao, H.; Wang, Y.; Chen, J.; Jiang, D.; Zhang, X.; Tian, Q.; and Wang, M. 2022. Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation. In Proceedings of the European Conference on Computer Vision Workshops(ECCVW)

    work page 2022

  7. [7]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248--255. Ieee

    2009

  8. [8]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    arXiv 2020

Show all 18 references
  1. [9]

    K.; P otka, S.; and Sitek, A

    Grzeszczyk, M. K.; P otka, S.; and Sitek, A. 2022. Multi-task Swin Transformer for Motion Artifacts Classification and Cardiac Magnetic Resonance Image Segmentation. In International Workshop on Statistical Atlases and Computational Models of the Heart, 409--417. Springer

    2022

  2. [10]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015. Deep Residual Learning for Image Recognition. CoRR, abs/1512.03385

    2015 arXiv

  3. [11]

    J.; Teney, D.; Kim, J.-H.; and Zhang, B.-T

    Hwang, I.; Lee, S.; Kwak, Y.; Oh, S. J.; Teney, D.; Kim, J.-H.; and Zhang, B.-T. 2022. SelecMix: Debiased Learning by Contradicting-pair Sampling. arXiv:2211.02291

    2022 arXiv

  4. [12]

    Y.; Arjovsky, M.; Pezeshki, M.; and Lopez-Paz, D

    Idrissi, B. Y.; Arjovsky, M.; Pezeshki, M.; and Lopez-Paz, D. 2022. Simple data balancing achieves competitive worst-group-accuracy. arXiv:2110.14503

    2022 arXiv

  5. [13]

    Kirichenko, P.; Izmailov, P.; and Wilson, A. G. 2023. Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations. arXiv:2204.02937

    2023 arXiv

  6. [14]

    Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    2021

  7. [15]

    Ronneberger, O.; Fischer, P.; and Brox, T. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention--MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part II...

    2015

  8. [16]

    W.; and Liang, P

    Sagawa, S.; Raghunathan, A.; Koh, P. W.; and Liang, P. 2020. An Investigation of Why Overparameterization Exacerbates Spurious Correlations. arXiv:2005.04345

    2020 arXiv

  9. [17]

    R.; Das, A.; Vedantam, R.; Cogswell, M.; Parikh, D.; and Batra, D

    Selvaraju, R. R.; Das, A.; Vedantam, R.; Cogswell, M.; Parikh, D.; and Batra, D. 2016. Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization. CoRR, abs/1610.02391

    2016 arXiv

  10. [18]

    Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and J \'e gou, H. 2021. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, 10347--10357. PMLR

    2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.