Exploiting bilateral symmetry in brain lesion segmentation

Kevin Raina; Tanya Schmah; Uladzimir Yahorau

arxiv: 1907.08196 · v1 · pith:2R74ZXC5new · submitted 2019-07-18 · 📡 eess.IV · cs.CV· cs.LG· q-bio.NC

Exploiting bilateral symmetry in brain lesion segmentation

Kevin Raina , Uladzimir Yahorau , Tanya Schmah This is my paper

Pith reviewed 2026-05-24 19:21 UTC · model grok-4.3

classification 📡 eess.IV cs.CVcs.LGq-bio.NC

keywords brain lesion segmentationbilateral symmetryreflective registrationCNNstroke lesionMRI segmentationISLES 2015

0 comments

The pith

Nonlinear reflective registration to an image's mirror improves CNN brain lesion segmentation by 9-13 Dice points.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that adding patches from the opposite hemisphere, located via nonlinear registration of a brain scan to its own reflection, boosts the accuracy of CNN-based lesion segmentation. This works because healthy brains are roughly symmetric, so the contralateral information supplies useful context when lesions create asymmetry. Two separate CNN architectures were tested on the ISLES 2015 SISS stroke dataset, each showing large gains when the symmetry features were included. A reader cares because lesion location and shape vary widely, making reliable automatic segmentation difficult, and the method adds complementary data without external templates.

Core claim

Nonlinear reflective registration aligns a neuroimage to a reflected version of itself to identify homologous voxels in the opposite hemisphere; patches around those voxels are then supplied as extra features to CNN segmentation models. On the SISS Training dataset from ISLES 2015, this symmetry augmentation raised Dice coefficients by 13 points for one architecture and 9 points for the other, outperforming both the unaugmented baselines and versions that used only affine reflective registration.

What carries the argument

Nonlinear reflective registration, which computes a deformation field mapping each voxel to its homologous location in the contralateral hemisphere by aligning the image to its mirror image.

If this is right

Affine reflective registration improves results modestly, but nonlinear registration produces substantially larger gains.
The symmetry features can be added to any existing segmentation algorithm.
The approach requires no external template and works directly on the input image.
Gains appear across two different CNN architectures for multimodal MRI stroke segmentation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same contralateral-feature trick could be tested on tumor segmentation where lesions are often more asymmetric.
If registration remains reliable, the method might allow smaller training sets by injecting a strong anatomical prior.
Performance on other modalities or non-stroke lesions would test whether the symmetry signal stays informative when pathology is more disruptive.

Load-bearing premise

Nonlinear registration to the reflected image can still locate accurate homologous voxels even when lesions break the brain's symmetry.

What would settle it

A dataset of manually labeled homologous landmark pairs in lesioned brains on which the nonlinear reflective registration produces large alignment errors, or a replication experiment on ISLES data where adding the contralateral patches yields no Dice gain.

Figures

Figures reproduced from arXiv: 1907.08196 by Kevin Raina, Tanya Schmah, Uladzimir Yahorau.

**Figure 1.** Figure 1: The quasi-symmetry property of the normal brain can be used to aid lesion segmentation. Subfigure (a) shows a lesion voxel (red) and its non-lesion (blue) mirror voxel (projected onto the same axial slice); (b) shows the voxel intensity plotted against the intensity of its mirror for a sample of 600 voxels taken from the same brain and equally divided between lesions and non-lesions; (c) and (d) show supe… view at source ↗

**Figure 2.** Figure 2: TwoPathCNN architecture, reproduced from Havaei et al. [4] with permission of the authors. Note that the final output is a 2 × 1 × 1 tensor for our application. 2.1 Baseline models TwoPathCNN and Wider2dSeg are convolutional neural networks. Both architectures take as input one or, as in the case of Kamnitsas’ 2D architectures, two stacks of four patches from different MRI modalities. The networks branch … view at source ↗

**Figure 3.** Figure 3: An example of different segmentations of Brain #8 of the SISS2015 Training set from the ISLES2015 Challenge. From left to right, the columns show segmentations produced by the TwoPathCNN, TwoPathCNN+LSymm and TwoPathCNN+NLSymm methods, and the ground truth. Moreover, nonlinearly registered symmetry features consistently produces higher Dice, Recall and Precision scores compared to linearly registered symm… view at source ↗

read the original abstract

Brain lesions, including stroke and tumours, have a high degree of variability in terms of location, size, intensity and form, making automatic segmentation difficult. We propose an improvement to existing segmentation methods by exploiting the bilateral quasi-symmetry of healthy brains, which breaks down when lesions are present. Specifically, we use nonlinear registration of a neuroimage to a reflected version of itself ("reflective registration") to determine for each voxel its homologous (corresponding) voxel in the other hemisphere. A patch around the homologous voxel is added as a set of new features to the segmentation algorithm. To evaluate this method, we implemented two different CNN-based multimodal MRI stroke lesion segmentation algorithms, and then augmented them by adding extra symmetry features using the reflective registration method described above. For each architecture, we compared the performance with and without symmetry augmentation, on the SISS Training dataset of the Ischemic Stroke Lesion Segmentation Challenge (ISLES) 2015 challenge. Using affine reflective registration improves performance over baseline, but nonlinear reflective registration gives significantly better results: an improvement in Dice coefficient of 13 percentage points over baseline for one architecture and 9 points for the other. We argue for the broad applicability of adding symmetric features to existing segmentation algorithms, specifically using nonlinear, template-free methods.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper gets solid Dice gains from nonlinear reflective registration on ISLES 2015 but never checks whether the registration itself stays accurate near lesions.

read the letter

The headline result is that adding contralateral patches via nonlinear reflective registration lifts Dice by 9-13 points over two baseline CNNs on the ISLES 2015 SISS training set. Affine registration helps less, so the nonlinear step appears to matter. The method is simple: register the image to its reflection, pull patches from the homologous locations, and feed them as extra channels. That concrete combination is not in the cited prior work, so the empirical demonstration counts as new for this task. The paper also shows the trick works on two different architectures and makes the case that it is template-free and therefore easy to drop onto other segmenters. Those are the clear positives. The main weakness is that nothing validates the registration quality inside or next to lesions. If the optimizer warps healthy tissue toward the lesion or produces large local errors, the added patches become misaligned or sample the lesion itself and the reported gains could be fragile. The abstract and methods give no landmark errors, no deformation-field visuals restricted to perilesional zones, and no ablation that separates registration accuracy from the CNN. The numbers also come without error bars, p-values, or repeated splits, so it is hard to judge stability. The dataset is a single public challenge set, which is fine for a first report but limits claims of broad applicability. This paper is aimed at people already running CNNs on brain lesion data who are looking for cheap symmetry-based features. A reader who wants to try the trick tomorrow will find enough detail to implement it and will see the size of the reported lift. It is worth sending to peer review because the gains are large enough to be interesting and the method is straightforward to reproduce, even though referees will almost certainly ask for registration diagnostics and statistical support.

Referee Report

3 major / 2 minor

Summary. The paper proposes to improve CNN-based multimodal MRI segmentation of brain lesions (stroke) by augmenting each voxel with features from a patch around its contralateral homologue, obtained via nonlinear reflective registration of the image to its left-right reflection. Two architectures are tested on the ISLES 2015 SISS training set; the authors report that nonlinear reflective registration yields Dice gains of 13 and 9 percentage points over the unaugmented baselines, while affine registration yields smaller gains. They conclude that the symmetry-augmented approach is broadly applicable.

Significance. If the registration step reliably recovers accurate homologous locations even in the presence of lesions, the method supplies a template-free way to inject bilateral symmetry information into existing segmentation pipelines. The reported Dice improvements are large enough to be practically interesting, but the manuscript supplies no direct evidence that the deformation fields remain faithful near lesions.

major comments (3)

[Abstract, §3] Abstract and §3 (Experiments): the headline claim that nonlinear reflective registration produces +13 / +9 Dice points rests on the untested assumption that the deformation field computed between I and reflect(I) still maps to accurate homologous voxels when a lesion breaks symmetry. No landmark-based registration error, no perilesional deformation-field QA, and no ablation that isolates registration quality from the downstream CNN are provided, so it is impossible to determine whether the added contralateral patches supply signal or noise.
[§2] §2 (Method): the description of the nonlinear registration step does not specify the similarity metric, regularization, or optimizer used, nor does it report any quantitative check that the resulting deformation field is invertible or that homologous patches avoid sampling lesioned tissue. These details are load-bearing for the central empirical claim.
[§3] §3 (Experiments): performance numbers are given without statistical tests, standard deviations across folds or runs, or details of the train/validation split on the SISS training set. Consequently the reported 13- and 9-point gains cannot be assessed for reliability.

minor comments (2)

The manuscript would benefit from a figure showing example deformation fields overlaid on lesioned regions to allow visual assessment of registration quality.
Implementation details (CNN hyperparameters, registration software and parameters, patch size) are referenced only at a high level; explicit values or a code repository would improve reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive comments. We address each of the major comments below and outline the revisions we plan to make.

read point-by-point responses

Referee: [Abstract, §3] Abstract and §3 (Experiments): the headline claim that nonlinear reflective registration produces +13 / +9 Dice points rests on the untested assumption that the deformation field computed between I and reflect(I) still maps to accurate homologous voxels when a lesion breaks symmetry. No landmark-based registration error, no perilesional deformation-field QA, and no ablation that isolates registration quality from the downstream CNN are provided, so it is impossible to determine whether the added contralateral patches supply signal or noise.

Authors: We agree that the manuscript does not include direct validation of the registration quality near lesions. The improvements are demonstrated through the segmentation performance gains, particularly the larger gains with nonlinear over affine registration. This provides indirect evidence that the method is capturing useful symmetry information. However, we acknowledge the value of additional QA and will add a limitations section discussing this point in the revised manuscript. revision: partial
Referee: [§2] §2 (Method): the description of the nonlinear registration step does not specify the similarity metric, regularization, or optimizer used, nor does it report any quantitative check that the resulting deformation field is invertible or that homologous patches avoid sampling lesioned tissue. These details are load-bearing for the central empirical claim.

Authors: We agree that these details were not included in the original submission. We will revise the method section to specify the similarity metric, regularization, optimizer, and any checks performed on the deformation fields. revision: yes
Referee: [§3] §3 (Experiments): performance numbers are given without statistical tests, standard deviations across folds or runs, or details of the train/validation split on the SISS training set. Consequently the reported 13- and 9-point gains cannot be assessed for reliability.

Authors: The experiments used the standard training set split from the ISLES 2015 challenge. We will add the split details and perform statistical tests where possible in the revision to better assess the reliability of the gains. revision: yes

Circularity Check

0 steps flagged

No circularity; purely empirical method comparison on public dataset

full rationale

The paper describes a registration-based feature augmentation technique and evaluates it by training two CNN architectures on the ISLES 2015 SISS training set, reporting Dice improvements for the augmented versions versus baselines. No equations, parameter fits, uniqueness theorems, or predictions are presented that reduce to the inputs by construction. All performance numbers are direct experimental outcomes from held-out evaluation, with no self-citation load-bearing the central claim and no renaming of known results as new derivations. The analysis is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that healthy brains are bilaterally quasi-symmetric and that this symmetry can be recovered via registration even when lesions are present.

axioms (1)

domain assumption Healthy brains exhibit bilateral quasi-symmetry that is disrupted by lesions.
Explicitly stated in the abstract as the foundation for using reflective registration to locate homologous voxels.

pith-pipeline@v0.9.0 · 5765 in / 1162 out tokens · 23738 ms · 2026-05-24T19:21:15.079812+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages

[1]

In: OSDI

Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghe- mawat, S., Irving, G., Isard, M., et al.: Tensorﬂow: a system for large-scale machine learning. In: OSDI. vol. 16, pp. 265–283 (2016)

work page 2016
[2]

Transac Med Imagins Penn Image Comput Sci Lab (2009)

Avants, B.B., Tustison, N.J., Song, G., Gee, J.C.: ANTS: Open-source tools for normalization and neuroanatomy. Transac Med Imagins Penn Image Comput Sci Lab (2009)

work page 2009
[3]

PIERS Proceedings, Stockholm, Sweden (2013)

Dvorak, P., Bartusek, K., Kropatsch, W.: Automated segmentation of brain tu- mour edema in ﬂair mri using symmetry and thresholding. PIERS Proceedings, Stockholm, Sweden (2013)

work page 2013
[4]

Medical image analysis 35, 18–31 (2017)

Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.M., Larochelle, H.: Brain tumor segmentation with deep neural net- works. Medical image analysis 35, 18–31 (2017)

work page 2017
[5]

In: Proceedings of the IEEE interna- tional conference on computer vision

He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectiﬁers: Surpassing human- level performance on imagenet classiﬁcation. In: Proceedings of the IEEE interna- tional conference on computer vision. pp. 1026–1034 (2015)

work page 2015
[6]

Medical image analysis 36, 61–78 (2017)

Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Eﬃcient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical image analysis 36, 61–78 (2017)

work page 2017
[7]

In: European Conference on Computer Vision

Loy, G., Eklundh, J.O.: Detecting symmetry and symmetric constellations of fea- tures. In: European Conference on Computer Vision. pp. 508–521. Springer (2006)

work page 2006
[8]

Medical image analysis 35, 250–269 (2017)

Maier, O., Menze, B.H., von der Gablentz, J., H¨ ani, L., Heinrich, M.P., Liebrand, M., Winzeck, S., Basit, A., Bentley, P., Chen, L., et al.: ISLES 2015 - a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Medical image analysis 35, 250–269 (2017)

work page 2015
[9]

Proceedings of MICCAI BRATS Challenge pp

Meier, R., Bauer, S., Slotboom, J., Wiest, R., Reyes, M.: Appearance-and context- sensitive features for brain tumor segmentation. Proceedings of MICCAI BRATS Challenge pp. 020–026 (2014)

work page 2014
[10]

In: Fourth International Conference on Machine Learning and Applications (ICMLA’05)

Schmidt, M., Levner, I., Greiner, R., Murtha, A., Bistritz, A.: Segmenting brain tumors using alignment-based features. In: Fourth International Conference on Machine Learning and Applications (ICMLA’05). pp. 6–pp. IEEE (2005)

work page 2005
[11]

In: 2017 IEEE International Con- ference on Image Processing (ICIP)

Shen, H., Zhang, J., Zheng, W.: Eﬃcient symmetry-driven fully convolutional net- work for multimodal brain tumor segmentation. In: 2017 IEEE International Con- ference on Image Processing (ICIP). pp. 3864–3868. IEEE (2017)

work page 2017
[12]

NeuroImage 13(6), 249 (2001) 10 Raina et al

Smith, S., Bannister, P.R., Beckmann, C., Brady, M., Clare, S., Flitney, D., Hansen, P., Jenkinson, M., Leibovici, D., Ripley, B., et al.: FSL: New tools for functional and structural brain image analysis. NeuroImage 13(6), 249 (2001) 10 Raina et al

work page 2001
[13]

Neuroinformatics 13(2), 209–225 (2015)

Tustison, N.J., Shrinidhi, K., Wintermark, M., Durst, C.R., Kandel, B.M., Gee, J.C., Grossman, M.C., Avants, B.B.: Optimal symmetric multimodal templates and concatenated random forests for supervised brain tumor segmentation (sim- pliﬁed) with antsr. Neuroinformatics 13(2), 209–225 (2015)

work page 2015
[14]

In: 2016 IEEE International Conference on Image Processing (ICIP)

Wang, Y., Katsaggelos, A.K., Wang, X., Parrish, T.B.: A deep symmetry convnet for stroke lesion segmentation. In: 2016 IEEE International Conference on Image Processing (ICIP). pp. 111–115. IEEE (2016)

work page 2016
[15]

Frontiers in neurology 9 (2018)

Winzeck, S., Hakim, A., McKinley, R., Pinto, J.A., Alves, V., Silva, C., Pisov, M., Krivov, E., Belyaev, M., Monteiro, M., et al.: ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI. Frontiers in neurology 9 (2018)

work page 2016

[1] [1]

In: OSDI

Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghe- mawat, S., Irving, G., Isard, M., et al.: Tensorﬂow: a system for large-scale machine learning. In: OSDI. vol. 16, pp. 265–283 (2016)

work page 2016

[2] [2]

Transac Med Imagins Penn Image Comput Sci Lab (2009)

Avants, B.B., Tustison, N.J., Song, G., Gee, J.C.: ANTS: Open-source tools for normalization and neuroanatomy. Transac Med Imagins Penn Image Comput Sci Lab (2009)

work page 2009

[3] [3]

PIERS Proceedings, Stockholm, Sweden (2013)

Dvorak, P., Bartusek, K., Kropatsch, W.: Automated segmentation of brain tu- mour edema in ﬂair mri using symmetry and thresholding. PIERS Proceedings, Stockholm, Sweden (2013)

work page 2013

[4] [4]

Medical image analysis 35, 18–31 (2017)

Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.M., Larochelle, H.: Brain tumor segmentation with deep neural net- works. Medical image analysis 35, 18–31 (2017)

work page 2017

[5] [5]

In: Proceedings of the IEEE interna- tional conference on computer vision

He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectiﬁers: Surpassing human- level performance on imagenet classiﬁcation. In: Proceedings of the IEEE interna- tional conference on computer vision. pp. 1026–1034 (2015)

work page 2015

[6] [6]

Medical image analysis 36, 61–78 (2017)

Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Eﬃcient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical image analysis 36, 61–78 (2017)

work page 2017

[7] [7]

In: European Conference on Computer Vision

Loy, G., Eklundh, J.O.: Detecting symmetry and symmetric constellations of fea- tures. In: European Conference on Computer Vision. pp. 508–521. Springer (2006)

work page 2006

[8] [8]

Medical image analysis 35, 250–269 (2017)

Maier, O., Menze, B.H., von der Gablentz, J., H¨ ani, L., Heinrich, M.P., Liebrand, M., Winzeck, S., Basit, A., Bentley, P., Chen, L., et al.: ISLES 2015 - a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Medical image analysis 35, 250–269 (2017)

work page 2015

[9] [9]

Proceedings of MICCAI BRATS Challenge pp

Meier, R., Bauer, S., Slotboom, J., Wiest, R., Reyes, M.: Appearance-and context- sensitive features for brain tumor segmentation. Proceedings of MICCAI BRATS Challenge pp. 020–026 (2014)

work page 2014

[10] [10]

In: Fourth International Conference on Machine Learning and Applications (ICMLA’05)

Schmidt, M., Levner, I., Greiner, R., Murtha, A., Bistritz, A.: Segmenting brain tumors using alignment-based features. In: Fourth International Conference on Machine Learning and Applications (ICMLA’05). pp. 6–pp. IEEE (2005)

work page 2005

[11] [11]

In: 2017 IEEE International Con- ference on Image Processing (ICIP)

Shen, H., Zhang, J., Zheng, W.: Eﬃcient symmetry-driven fully convolutional net- work for multimodal brain tumor segmentation. In: 2017 IEEE International Con- ference on Image Processing (ICIP). pp. 3864–3868. IEEE (2017)

work page 2017

[12] [12]

NeuroImage 13(6), 249 (2001) 10 Raina et al

Smith, S., Bannister, P.R., Beckmann, C., Brady, M., Clare, S., Flitney, D., Hansen, P., Jenkinson, M., Leibovici, D., Ripley, B., et al.: FSL: New tools for functional and structural brain image analysis. NeuroImage 13(6), 249 (2001) 10 Raina et al

work page 2001

[13] [13]

Neuroinformatics 13(2), 209–225 (2015)

Tustison, N.J., Shrinidhi, K., Wintermark, M., Durst, C.R., Kandel, B.M., Gee, J.C., Grossman, M.C., Avants, B.B.: Optimal symmetric multimodal templates and concatenated random forests for supervised brain tumor segmentation (sim- pliﬁed) with antsr. Neuroinformatics 13(2), 209–225 (2015)

work page 2015

[14] [14]

In: 2016 IEEE International Conference on Image Processing (ICIP)

Wang, Y., Katsaggelos, A.K., Wang, X., Parrish, T.B.: A deep symmetry convnet for stroke lesion segmentation. In: 2016 IEEE International Conference on Image Processing (ICIP). pp. 111–115. IEEE (2016)

work page 2016

[15] [15]

Frontiers in neurology 9 (2018)

Winzeck, S., Hakim, A., McKinley, R., Pinto, J.A., Alves, V., Silva, C., Pisov, M., Krivov, E., Belyaev, M., Monteiro, M., et al.: ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI. Frontiers in neurology 9 (2018)

work page 2016