A unified representation network for segmentation with missing modalities

Jens Sj\"olund; Jonas Adler; Kenneth Lau

REVIEW 3 major objections 5 minor 1 cited by

A unified representation network for segmentation with missing modalities

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A unified representation network, trained with modality dropout, segments brain tumors better than a standard U-net when MRI sequences are missing at test time.

desk verdict A pragmatic, readable test of modality dropout plus a fused-representation URN on BRATS, but the central advantage over the simpler baseline rests on small single-run Dice gaps with no error bars and no HeMIS comparison. read the letter →

arxiv 1908.06683 v1 pith:G6RPATXB submitted 2019-08-19 cs.CV

Kenneth Lau , Jonas Adler , Jens Sjölund This is my paper · ORCID

classification cs.CV

keywords missingmodalitiesmodalitydropoutunifiedrepresentationnetworkmultimodalMRIsegmentationbraintumorlearningunsupervisedpre-training

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies a realistic failure of deep segmentation networks: at test time, one or more of the MRI sequences used during training may be missing. The authors propose two compatible fixes: modality dropout, a training-time trick that randomly zeros entire input channels, and a unified representation network (URN), which encodes each available modality into a common latent image and fuses the encodings before a downstream segmenter. On a public four-sequence brain tumor segmentation benchmark, a standard U-net trained with modality dropout becomes reasonably robust to missing inputs, and the same segmenter performs better when trained on the URN's unified representation, with the highest whole-tumor overlap score for every input combination except one. The paper also shows that unsupervised pre-training on a second dataset with only partially overlapping modalities can improve segmentation of some tumor regions.

What carries the argument

The central object is the URN's fusion module: a generalized f-mean, here the simple mean, that combines the outputs of modality-specific U-net encoders into one representation whose magnitude does not depend on how many modalities are available. A voxel-wise variance penalty, along with batch normalization that fixes channel statistics, is what pushes the per-modality encoders to produce aligned representations rather than identity mappings. Modality dropout, sampling the number of dropped channels from a truncated geometric distribution, is applied during URN training so the fusion learns from variable subsets of inputs.

What would settle it

Retrain the URN with the voxel-wise variance penalty removed and every other design choice fixed; if Dice scores on missing-modality inputs stay the same, the unified representation is not what carries the improvement, and the extra encoder parameters would be the likely cause.

Watch

Extended reading notes

Core claim

The central claim is that the Unified Representation Network (URN) maps a variable number of input MRI modalities into a shared latent representation, built as a generalized f-mean, here the plain average of per-modality encoder outputs, and that a standard segmentation decoder trained on this representation outperforms both a plain U-net and a U-net trained with modality dropout when modalities are missing. On the evaluated brain tumor data, the URN achieved the highest whole-tumor Dice score for all combinations of available inputs except one, and the authors state that the same network works even better when trained on the unified representation than with modality dropout alone. A further claim is that unsupervised pre-training on a second dataset with partially overlapping modalities, followed by fixed-weight segmentation, improves the segmentation of enhancing tumor and tumor core regions while leaving whole-tumor performance comparable.

Load-bearing premise

The result rests on the assumption that the penalty pushing each modality's encoder to produce similar outputs yields a common representation that retains the information segmentation needs, rather than merely aligning the encoders for image reconstruction.

Editorial extensions

If this is right

A single trained URN segmenter can be queried with any subset of the four MRI sequences, avoiding the combinatorial explosion of training one network per input combination.
Modality dropout makes a standard U-net reasonably robust to missing modalities and, on this benchmark, also improves performance when all modalities are present, suggesting it acts as a regularizer that encourages feature sharing.
Because the fusion is an intensive property, the architecture handles a variable number of input modalities without changing the network.
Unsupervised pre-training on a second dataset with only partially overlapping modalities can improve some tumor-region segmentations, indicating that unlabeled data from other acquisitions can be exploited.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the fusion mechanism is modality-agnostic, so the URN design should transfer to other multimodal segmentation settings, such as CT and MRI or multi-sequence cardiac imaging, with different modality counts; the paper tests only four MRI sequences.
Editorial inference: the reported benefit of pooled pre-training is region-dependent, so a practical rule would need to decide when extra pre-training helps rather than assuming it always does.
Editorial inference: a parameter-matched control with per-modality encoders whose outputs are combined without the variance penalty would isolate whether the unified representation, rather than the larger encoder ensemble, drives the improvement; the paper does not include this control.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: this is a clean, honest empirical paper on a real clinical problem — missing MRI modalities at segmentation time. What is genuinely new is not the architecture (it is Chartsias et al.'s URN, applied to segmentation) but the specific comparison: modality dropout alone versus URN plus modality dropout, and the HCP pre-training transfer. Those measurements are new, and the HCP-pretraining results, buried in the supplementary, are actually the most interesting part: they lift tumor core by several Dice points and help image synthesis from single modalities.

What the paper does well: the writing is clear, the method section is reproducible enough, and the authors are honest about limitations — they explicitly say the URN was proposed before for synthesis, they note the T1-only exception where baseline+MD beats URN+MD, and they flag the missing baseline T1 leaderboard entry. That counts for something. Modality dropout itself is shown to be a surprisingly effective baseline, which is a useful sanity check for the field.

Where it gets soft: the load-bearing claim that URN+MD beats baseline+MD on whole tumor is supported only by single-run leaderboard scores. Most gaps are 1–3 Dice points, which on BRATS U-net variants is squarely within run-to-run noise. There are no error bars, no repeated seeds, no significance test. The URN also has far more parameters (four encoder U-nets plus fusion), so even a real gap would not prove the unified representation is the cause. The paper omits HeMIS, the closest prior method that already fuses modality-specific encoder outputs for segmentation; without that baseline, the specific contribution of the f-mean fusion is unproven. One text claim — "highest Dice for all combinations except one" — is contradicted by their own Table 1 for tumor core, where baseline+MD wins several combinations. That inconsistency is minor in itself, but it does not inspire confidence in the table's provenance.

I think the central mechanism claim is plausible but under-evidenced. The paper would be much stronger with repeated runs, error bars, a HeMIS comparison, and code release. The HCP pre-training result deserves more prominence than a supplementary aside.

Who this is for: anyone working on missing-modality segmentation or multi-dataset representation learning. It is worth a serious referee — the question is real, the experiments are reproducible in principle, and the limitations are mostly acknowledged. I lean conditional accept with major revision, not desk reject.

Best, [You]

Referee Report

3 major / 5 minor

Summary. The paper addresses segmentation of brain tumors from MR images when some input modalities are missing at test time. It describes modality dropout (randomly zeroing full input channels during training) and a Unified Representation Network (URN) with modality-specific U-net encoders, a fusion step based on the f-mean (Eq. (2)), and a variance-based regularization to encourage alignment of the encoder outputs. The URN can be pre-trained in an unsupervised way on BRATS and HCP, and a segmentation head is trained on the fused representation. Experiments on the BRATS 2018 validation set compare a plain U-net baseline, baseline with modality dropout, URN with modality dropout, and URN pre-trained on HCP plus BRATS, reporting Dice scores for enhancing tumor, whole tumor, and tumor core across all 15 non-empty modality subsets. The main reported result is that the URN with modality dropout gives higher whole-tumor Dice than the baseline with modality dropout for most modality combinations, with one exception (T1-only).

Significance. If the empirical claim is reproducible, the paper would provide a practical method for handling missing modalities without training one network per modality subset, and the cross-dataset pre-training aspect is interesting. The use of the independent BRATS validation leaderboard and the public HCP dataset is a strength, as is the explicit reporting of the T1-only counterexample. However, the main evidence is a single set of leaderboard scores with no error bars or significance testing, and the closest prior method (HeMIS) is not compared. The architectural novelty over Chartsias et al. and van Tulder is incremental, and no code is released, which limits reproducibility. The paper's contribution therefore rests on a thin empirical base that needs strengthening.

major comments (3)

[Section 3.2 and Supplementary Table 1] The central claim is not uniformly supported for all tumor regions and lacks uncertainty quantification. For whole tumor the URN+MD is higher than Baseline+MD in 14 of 15 input combinations, with T1-only as the exception (WT 50.4 vs 51.9), but for tumor core the URN+MD is lower in several rows, for example with all modalities (TC 71.8 vs 73.9) and with T1/T1c/T2 (TC 69.5 vs 71.0). The abstract's statement that 'the same network works even better' is therefore too broad. Moreover, all scores come from a single submission to the BRATS validation leaderboard; with differences of 1–3 points, run-to-run variance could change the ranking. Please report multiple runs with means and standard deviations or confidence intervals, and restrict the robustness claim to the regions and input sets for which it actually holds.
[Section 1 and Section 3.2 (missing HeMIS comparison)] HeMIS [4], which the paper itself cites as the notable prior exception, also uses modality-specific encoders and a fusion operation for segmentation with missing modalities. Without an experimental comparison to HeMIS under identical training conditions, the reported URN advantage over a plain U-net with modality dropout could come from the larger parameter count or from the specific pre-training scheme rather than from the f-mean fusion and variance regularization. Adding a HeMIS baseline, and ideally a parameter-matched ablation, is necessary to support the claim that the unified representation is responsible for the improvement.
[Section 2.3.2 and Section 2.3.3] The paper does not verify that the fused representation is actually unified or that the fusion step is the source of the improvement. The variance loss in Section 2.3.2 encourages the encoders' outputs to be similar on the reconstruction task, but there is no evidence that the representation used by the segmentation head is modality-invariant or that it retains the task-relevant information. For example, training the segmentation head on concatenated (zero-filled) encoder outputs, or measuring the distance between encoder outputs for different input sets, would test the mechanism. This concern is load-bearing for the paper's interpretation of the architecture, even if not for the raw empirical comparison.

minor comments (5)

[Section 2.3.1] The phrase 'batch normalization with fixed parameters' is unclear; please state whether the batch-normalization statistics are frozen from pre-training or recomputed during segmentation training.
[Section 2.2, Eq. (1)] Equation (1) uses N in the denominator while the text defines N_max; please use a single symbol consistently.
[Section 3.2] The text contains a typo, 'enhancing tumor anc tumor core', which should read 'enhancing tumor and tumor core'.
[Supplementary Table 1] The note 'Results of T1 using Baseline model is unable to be retrieved from the leaderboard portal' is awkwardly phrased and the missing entry should be explained more clearly, for example by stating that the leaderboard did not return a score for that configuration.
[Reproducibility] No mention is made of code or model release; providing these would substantially help other researchers verify the reported leaderboard scores.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are empirical comparisons on an external benchmark, not derivations from fitted inputs.

full rationale

This paper does not contain a derivation chain in which a predicted quantity is equivalent to an input by construction. The central claim is empirical: a Unified Representation Network trained with modality dropout achieves higher BRATS whole-tumor Dice scores than a standard U-net trained with modality dropout for most missing-modality input combinations. All models are evaluated on the independent BRATS validation leaderboard, and the reported numbers come from that external evaluation rather than from any equation in the paper. The f-mean fusion rule (Eq. (2)) is an architectural definition, not a derived result, and the claimed segmentation benefit is supported by measured Dice scores rather than by Eq. (2) alone. The unsupervised pretraining on HCP and downstream segmentation on BRATS are separate stages, so no fitted parameter is later renamed as a prediction. The cited works [3] and [13] are prior external publications, not self-citations, and they are used only for architectural details such as decoder design, channel count, and batch-normalization standardization; they do not carry the load of the empirical conclusion. The limitations noted in the manuscript and by a skeptical reader, such as lack of repeated seeds, lack of error bars, the missing T1 baseline entry, and the absence of a HeMIS comparison, concern statistical robustness and baseline coverage. These are legitimate concerns about evidence quality or correctness risk, but they are not circularity. No step in the paper reduces to its own inputs, and no self-citation chain forces the claimed result.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claims rest on empirical assumptions about dataset reliability and transfer of self-supervised representations, not on mathematical derivations. The listed free parameters are hand-set hyperparameters for the networks; none is derived or presented as a prediction.

free parameters (6)

Number of channels in unified representation = 16
Chosen based on prior MR synthesis work [3]; not optimized in this paper.
Modality dropout parameter theta for segmentation = 0.5
Controls the truncated geometric distribution in Eq. (1) when training segmentation models; set by hand.
Modality dropout parameter theta for pre-training = 0.8
Controls dropout during URN pre-training; set by hand.
Variance regularization weight = 1e-4
Weights the similarity term in the URN loss; set by hand.
Adam learning rate for segmentation = 1e-4
Default optimizer setting used for all segmentation models.
Adam learning rate for pre-training = 3e-5
Used for URN pre-training; set by hand.

assumptions (4)

domain assumption BRATS 2018 labels and the online validation leaderboard provide a reliable ranking of segmentation methods.
All conclusions are based on Dice scores from the BRATS portal; no manual verification of labels or leaderboard was performed in the paper.
domain assumption A 2D U-net with batch normalization and leaky ReLU is a representative segmentation backbone.
The paper fixes the architecture and only varies handling of missing modalities, so claims are about these mechanisms on this backbone.
ad hoc to paper The f-mean fusion plus variance loss produces a representation that retains task-relevant information for segmentation.
The paper relies on this to justify the URN's advantage, but offers no analysis showing which task information is preserved in the fused representation.
domain assumption Unsupervised pre-training on healthy adult HCP scans transfers to tumor segmentation on BRATS.
The reported enhancing tumor and tumor core improvements from HCP pre-training assume the healthy-brain representation is useful for tumor structures.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A unified representation network for segmentation with missing modalities." pith.science (2026). https://pith.science/paper/G6RPATXB

@misc{pith2026190806683,
  author       = {Pith},
  title        = {Pith review of: A unified representation network for segmentation with missing modalities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G6RPATXB}},
  note         = {Machine review of arXiv:1908.06683}
}

read the original abstract

Over the last few years machine learning has demonstrated groundbreaking results in many areas of medical image analysis, including segmentation. A key assumption, however, is that the train- and test distributions match. We study a realistic scenario where this assumption is clearly violated, namely segmentation with missing input modalities. We describe two neural network approaches that can handle a variable number of input modalities. The first is modality dropout: a simple but surprisingly effective modification of the training. The second is the unified representation network: a network architecture that maps a variable number of input modalities into a unified representation that can be used for downstream tasks such as segmentation. We demonstrate that modality dropout makes a standard segmentation network reasonably robust to missing modalities, but that the same network works even better if trained on the unified representation.

Figures

Figures reproduced from arXiv: 1908.06683 by the authors.

**Figure 2.** Dice scores of “whole tumor” on BRATS with different modalities. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Examples of tumor segmentations on BRATS given different combinations of inputs, overlaid on T1c. Red: [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Examples of image reconstruction and synthesis given different combinations of inputs only trained on [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Examples of image reconstruction and synthesis given different combinations of inputs while pre-trained [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
cs.CV 2026-07 conditional novelty 6.0 of 10

Prototype-conditioned Mixture-of-Experts synthesizes missing modalities in federated learning and beats prior methods on heterogeneous chest X-ray clients without public data.

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages · cited by 1 Pith paper

[4]

In: MICCAI

Havaei, M., et al.: HeMIS: Hetero-modal image segmentation. In: MICCAI. pp. 469–477. Springer (2016)

work page 2016
[1]

ICLR (2019)

Ardizzone, L., et al.: Analyzing inverse problems with invertible neural networks. ICLR (2019)

work page 2019
[2]

Scientiﬁc data 4, 170117 (2017)

Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientiﬁc data 4, 170117 (2017)

work page 2017
[3]

IEEE transactions on medical imaging (2017)

Chartsias, A., et al.: Multimodal MR synthesis via modality-invariant latent representation. IEEE transactions on medical imaging (2017)

work page 2017
[5]

arXiv:1411.2539 (2014)

Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multimodal neural language models. arXiv:1411.2539 (2014)

arXiv 2014
[6]

Medical image analysis 42, 60–88 (2017)

Litjens, G., et al.: A survey on deep learning in medical image analysis. Medical image analysis 42, 60–88 (2017)

work page 2017
[7]

IEEE transactions on medical imaging 34(10), 1993–2024 (2015)

Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE transactions on medical imaging 34(10), 1993–2024 (2015)

work page 2015
[8]

IJROBP 82(1), 368–378 (2012)

Nelms, B.E., et al.: Variations in the contouring of organs at risk: test case from a patient with oropharyngeal cancer. IJROBP 82(1), 368–378 (2012)

work page 2012

Show all 16 references

[9]

IEEE TPAMI 38(8), 1692–1706 (2016)

Neverova, N., et al.: ModDrop: adaptive multi-modal gesture recognition. IEEE TPAMI 38(8), 1692–1706 (2016)

2016
[10]

In: ICML

Ngiam, J., et al.: Multimodal deep learning. In: ICML. pp. 689–696 (2011)

2011
[11]

Available online (2019)

Radford, A., et al.: Language models are unsupervised multitask learners. Available online (2019)

2019
[12]

In: MICCAI

Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234–241. Springer (2015)

2015
[13]

IEEE transactions on medical imaging 38(2), 638–648 (2019)

van Tulder, G., de Bruijne, M.: Learning cross-modality representations from multi-modal images. IEEE transactions on medical imaging 38(2), 638–648 (2019)

2019
[14]

IEEE transactions on medical imaging 29(6), 1310 (2010)

Tustison, N.J., et al.: N4ITK: improved N3 bias correction. IEEE transactions on medical imaging 29(6), 1310 (2010)

2010
[15]

Neuroimage 80, 62–79 (2013)

Van Essen, D.C., et al.: The WU-Minn human connectome project: an overview. Neuroimage 80, 62–79 (2013)

2013
[16]

In: CVPR

Zhang, Z., Yang, L., Zheng, Y .: Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network. In: CVPR. pp. 9242–9251 (2018) 6 Supplementary material Experiments & Results The following tables demonstrate the detailed ...

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [4]

In: MICCAI

Havaei, M., et al.: HeMIS: Hetero-modal image segmentation. In: MICCAI. pp. 469–477. Springer (2016)

work page 2016

[2] [1]

ICLR (2019)

Ardizzone, L., et al.: Analyzing inverse problems with invertible neural networks. ICLR (2019)

work page 2019

[3] [2]

Scientiﬁc data 4, 170117 (2017)

Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientiﬁc data 4, 170117 (2017)

work page 2017

[4] [3]

IEEE transactions on medical imaging (2017)

Chartsias, A., et al.: Multimodal MR synthesis via modality-invariant latent representation. IEEE transactions on medical imaging (2017)

work page 2017

[5] [5]

arXiv:1411.2539 (2014)

Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multimodal neural language models. arXiv:1411.2539 (2014)

arXiv 2014

[6] [6]

Medical image analysis 42, 60–88 (2017)

Litjens, G., et al.: A survey on deep learning in medical image analysis. Medical image analysis 42, 60–88 (2017)

work page 2017

[7] [7]

IEEE transactions on medical imaging 34(10), 1993–2024 (2015)

Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE transactions on medical imaging 34(10), 1993–2024 (2015)

work page 2015

[8] [8]

IJROBP 82(1), 368–378 (2012)

Nelms, B.E., et al.: Variations in the contouring of organs at risk: test case from a patient with oropharyngeal cancer. IJROBP 82(1), 368–378 (2012)

work page 2012