Pith. sign in

REVIEW 5 major objections 6 minor 19 references

3D U$^2$-Net: A 3D Universal U-Net for Multi-Domain Medical Image Segmentation

T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single 3D U-Net with tiny per-domain adapters matches five separately trained segmentation models using about 1% of the parameters.

desk verdict The parameter-efficiency story is solid and the architecture is a genuine first for extendable 3D multi-domain segmentation, but the claimed accuracy parity breaks on Pancreas and the paper needs a revision that takes that drop seriously. read the letter →

arxiv 1909.06012 v1 pith:LSPQPBIS submitted 2019-09-04 eess.IV cs.CV

Chao Huang , Hu Han , Qingsong Yao , Shankuan Zhu , S. Kevin Zhou This is my paper · ORCID
classification eess.IVcs.CV
keywords multi-domainlearningmedicalimagesegmentation3DU-Netseparableconvolutiondomainadapterparameterefficiencyorgandeepsupervision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single 3D U-Net can segment organs across several imaging domains at nearly the accuracy of separately trained models while using about 1% of the parameters. The universal network replaces every standard 3x3x3 convolution with a domain adapter that splits the computation into per-domain channel-wise filters and one shared pointwise filter, so that what is common to all modalities lives in shared weights and what is specific to each modality stays tiny. On five public organ segmentation tasks (heart, liver, hippocampus, prostate, pancreas) the universal model reaches a mean Dice of 83.10, close to the 84.28 of independent models, with 1.7 million parameters instead of 126.7 million. It also adapts to a new spleen domain by adding only 0.1 million parameters, producing a Dice of 91.60 without retraining the shared weights. If correct, this makes a single extendable network, rather than a zoo of per-task models, a practical option for multi-organ medical image segmentation.

What carries the argument

The load-bearing mechanism is the domain adapter built from separable convolution. A standard $3\times3\times3$ convolution with $C$ input channels and $C'$ output channels costs $27CC'$ weights per filter set; the adapter replaces it with $C$ channel-wise $3\times3\times3$ filters (one per input channel, domain-specific, costing $27CT$ for $T$ domains) followed by a pointwise $1\times1\times1$ convolution shared by all domains (costing $CC'$). This keeps the total parameter growth additive in the number of domains instead of multiplicative. The adapter is inserted at every stride-1 $3\times3\times3$ convolution of a 3D U-Net that also has residual connections, skip connections, and a deep supervision branch, and the universal model is trained end-to-end with a Lovász-Softmax plus focal loss while batches are drawn from all domains in rotation.

What would settle it

Train the universal model on a domain whose segmentation depends on combining multiple input channels in a domain-specific way, such as fusing T2 and ADC MRI sequences into one prostate map, with the shared pointwise weights frozen and only new channel-wise adapters added; if Dice falls far below an independently trained full model on that same data, the spatial-versus-cross-channel split is refuted.

Watch

Extended reading notes

Core claim

The central claim is that a universal 3D U-Net, with domain adapters replacing stride-1 standard convolutions, can handle multiple medical segmentation domains simultaneously with accuracy comparable to per-task models. Each domain adapter contains only a small set of domain-specific channel-wise 3x3x3 filters per layer, while a pointwise 1x1x1 convolution is shared by all domains; joint training in a round-robin schedule makes the shared pointwise weights learn cross-channel correlations that transfer across CT and MRI tasks. The measured result is a universal model with 1.7 million parameters that scores 83.10 mean Dice across five base domains, versus 84.28 for five independent models totaling 126.7 million parameters, and the per-domain adapters also recover some of the loss a fully shared model suffers on the harder pancreas task. Extending to a new domain amounts to training a fresh set of channel-wise filters and input/output layers, about 0.1 million parameters, and the paper reports 91.60 Dice on the new spleen domain against 90.67 for a fully shared model. The intended takeaway is that segmentation models need not be rebuilt from scratch for each organ or modality.

Load-bearing premise

The load-bearing premise is that every imaging domain differs mainly in spatial texture within each feature channel, while the rules for combining channels are the same across domains; if a domain needs a genuinely different cross-channel interaction, the tiny per-domain filters cannot represent it and the accuracy advantage would disappear.

Editorial extensions

If this is right

  • Deploying the universal model means storing one shared backbone for all supported organs; each new domain adds only a small adapter, roughly 0.1 million parameters, instead of a full 30.7-million-parameter network.
  • Training time and data requirements per new task drop sharply, because a new domain only updates the domain-specific filters while the shared pointwise weights stay frozen.
  • Because domains are trained jointly, improvements or data from one domain can transfer to others, which the paper identifies as the reason prostate peripheral-zone and transition-zone scores exceed the independent models.
  • The same universal adapter design is proposed as a route to other multi-domain applications, not just organ segmentation, since the shared pointwise weights carry a reusable cross-channel representation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the channel-wise/pointwise split transfers to other modalities as cleanly as it does across CT and MRI, the same adapter scheme should work for 2D segmentation and for detection, where the shared pointwise layers could be reused across datasets with per-dataset spatial filters only.
  • Beyond the paper: the parameter economy suggests a distribution model the paper does not discuss, where a central universal backbone is kept in one place and only 0.1-million-parameter adapters are shipped to each clinical site or embedded device.
  • Beyond the paper: the strongest stress test would be adding a domain with fundamentally different image statistics, such as ultrasound or histopathology; the paper's five base domains are all CT or MRI, so they may be more alike than the universal claim requires.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes 3D U2-Net, a single 3D U-Net architecture for multiple medical image segmentation domains. Standard 3x3x3 convolutions are replaced with separable convolutions consisting of domain-specific channel-wise filters and shared pointwise filters, called domain adapters. The model is trained jointly on five Medical Segmentation Decathlon base domains (heart, liver, hippocampus, prostate, pancreas) using a combination of Lovász-Softmax and focal loss, and then adapted to a sixth, new domain (spleen) by adding channel-wise filters. The authors report a mean Dice of 83.10 for the universal model versus 84.28 for per-domain independent models, with about 1.7M parameters versus 126.7M for the independent models, and they release the code. They conclude that the universal network competes with traditional models in accuracy while using about 1% of the parameters and that it can be adapted to a new domain without sacrificing base-domain performance.

Significance. If the claims held, this would be a useful contribution to multi-domain medical image segmentation: a single compact model with public code, a two-orders-of-magnitude reduction in parameters, and a simple extension mechanism. The paper is among the first to transfer the multi-domain learning idea from image classification to 3D segmentation, and the public code is a concrete strength. However, the evidence in the manuscript is not strong enough to support the headline claims: one of the five base domains shows a large accuracy drop, no uncertainty quantification is provided, and the claim about preserving base-domain performance after extension is not actually measured. The contribution is promising but needs substantially stronger experimental support.

major comments (5)
  1. [Table 2, Base05 Pancreas] The central claim that the universal model is 'capable of competing with traditional models in terms of segmentation accuracy' is contradicted by the paper's own results on the pancreas: the independent model achieves 78.78 Dice while the universal model achieves 62.08, an absolute drop of 16.7 points, whereas all other base domains differ by at most 1.28 points. Since Pancreas is a base domain used during joint training, this is not an extrapolation failure; it is a failure on a core task. The average of 83.10 versus 84.28 hides this failure. The authors should either report per-domain significance, explain why this domain fails, or revise the central claim to explicitly exclude or qualify this domain.
  2. [Section 3, Implementation details and Table 2] No error bars, repeated runs, or statistical significance tests are reported anywhere in the manuscript. Single-run Dice differences of less than one point (e.g., 93.26 vs 91.98 on Heart) are used to support the claim of comparable accuracy. At minimum, each configuration should be run with multiple seeds and the mean and standard deviation reported; ideally, paired significance tests should be provided per domain so that readers can judge whether the observed differences are meaningful.
  3. [Section 3, Preprocessing and Implementation details] The comparison between independent and universal models is not controlled: independent models use per-domain patch sizes and numbers of downsampling operations adapted to image size, while the shared and universal models use a fixed 128x128x128 patch and six downsampling operations. This protocol difference could partly explain the accuracy gap on pancreas and other domains. The authors should either use matched training configurations for all models or justify the discrepancy and analyze its effect on the results.
  4. [Section 3, Table 3(b) and Abstract] The abstract and conclusion claim that adapting the universal network to a new domain does not sacrifice performance in the domains used to learn the shared parameterization. However, no post-adaptation results on the base domains are reported; only the new spleen Dice is shown in Table 3(b). This claim is therefore unsupported by the experiments. The authors should measure and report base-domain accuracy after adding the new domain, or remove the claim.
  5. [Section 2.2] The core architectural assumption—that domain-specific spatial correlations can be captured by channel-wise convolutions while cross-channel correlations can be shared via pointwise convolutions—is asserted but not validated. The large pancreas drop is consistent with this factorization being too restrictive for that domain. An ablation that, for example, adds a small number of per-domain pointwise or residual parameters, or that analyzes the learned channel-wise filters per domain, would turn this assumption into a tested claim rather than an untested premise.
minor comments (6)
  1. [Section 2.4, Eq. (2)] The relative weighting of the Lovász-Softmax and focal loss terms is not specified; please state the combination coefficients and whether they are tuned per domain or shared.
  2. [Table 3(a)] The column labeled 'Ratio' is ambiguous: it is unclear whether the values are relative to the shared model or to the independent models, especially since the text refers to 'about 1% of the independent models' while the table shows 0.06x. Please clarify the reference model for each ratio.
  3. [Section 3, Implementation details] Use 'PyTorch' and 'Adam' consistently; the manuscript uses 'Pytorch' and 'ADAM'.
  4. [Section 2.4] 'Lovsz-Softmax' should be spelled 'Lovász-Softmax'.
  5. [Section 1] The phrase 'we claim to note' is awkward and should be rephrased.
  6. [Table 2] For the Prostate domain, the table lists four structures (Anterior, Posterior, PZ, TZ); please state explicitly how the per-domain mean Dice is computed, e.g., macro-averaging over structures and patients.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the universal-network Dice scores are measured on held-out Medical Segmentation Decathlon data, and the separable-convolution prior is stated as an explicit assumption inherited from external work [5], not as a derived prediction.

full rationale

The paper's derivation chain is self-contained at the level of the claims it actually makes. The architecture section (2.2) explicitly states the domain-adapter factorization as an assumption: 'assuming that images from different domains have domain-specific spatial correlations which can be probed with channel-wise convolution while also share cross-channel correlations which can be modeled with pointwise convolution.' The parameter counts in Table 3 are arithmetic consequences of that architecture, not fitted values. The Dice results in Tables 2 and 3(b) are evaluation scores on held-out 20% test portions of the public Medical Segmentation Decathlon datasets, so the central comparison is an external measurement rather than an identity. The authors' prior work [9] is cited only as background on multi-task segmentation in the introduction and does not supply any load-bearing theorem or fitted constant. The citation to [5] is external, and the paper explicitly differs from [5] in joint training and extensibility. No equation in the paper reduces a prediction to an input by construction, and no fitted parameter is renamed as a prediction. The large Pancreas Dice drop (78.78 independent vs 62.08 universal in Table 2) is a substantive empirical failure of the modeling assumption, but it is evidence against the accuracy claim, not circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical, and the only substantive non-empirical premise is the separable-convolution factorization assumption. Training hyperparameters are hand-set but standard. No new entities such as particles, forces, or conserved quantities are introduced.

free parameters (2)
  • Learning rate and weight decay = 3e-4 and 1e-5
    Hand-set ADAM settings in Section 3. Standard choices, not fitted to the benchmark, but they affect all reported Dice scores.
  • Patch size and downsampling count for shared/universal model = 128x128x128 with 6 downsampling operations
    Chosen in Section 3 to fit GPU memory. Independent models use per-domain patch sizes, which creates a mild asymmetry in the comparison.
assumptions (3)
  • domain assumption Images from different domains have domain-specific spatial correlations that can be probed with channel-wise convolution, while cross-channel correlations are shared and can be modeled with pointwise convolution.
    Stated in the abstract and Section 2.2. If false, the adapter design loses its justification and cannot represent what a domain needs.
  • domain assumption The five Medical Segmentation Decathlon datasets plus Spleen, with the described cropping, resampling, clipping, and normalization, are adequate external benchmarks for the claim.
    Section 3 relies on these public datasets and preprocessing choices to support the empirical comparison.
  • domain assumption A standard 3D U-Net/V-Net architecture is an appropriate base network for all domains.
    Section 2.3 builds the universal model on this base; the choice of base architecture could influence whether the universal adapters provide a fair comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3D U$^2$-Net: A 3D Universal U-Net for Multi-Domain Medical Image Segmentation." pith.science (2026). https://pith.science/paper/LSPQPBIS

@misc{pith2026190906012,
  author       = {Pith},
  title        = {Pith review of: 3D U$^2$-Net: A 3D Universal U-Net for Multi-Domain Medical Image Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LSPQPBIS}},
  note         = {Machine review of arXiv:1909.06012}
}
abstract

Fully convolutional neural networks like U-Net have been the state-of-the-art methods in medical image segmentation. Practically, a network is highly specialized and trained separately for each segmentation task. Instead of a collection of multiple models, it is highly desirable to learn a universal data representation for different tasks, ideally a single model with the addition of a minimal number of parameters steered to each task. Inspired by the recent success of multi-domain learning in image classification, for the first time we explore a promising universal architecture that handles multiple medical segmentation tasks and is extendable for new tasks, regardless of different organs and imaging modalities. Our 3D Universal U-Net (3D U$^2$-Net) is built upon separable convolution, assuming that {\it images from different domains have domain-specific spatial correlations which can be probed with channel-wise convolution while also share cross-channel correlations which can be modeled with pointwise convolution}. We evaluate the 3D U$^2$-Net on five organ segmentation datasets. Experimental results show that this universal network is capable of competing with traditional models in terms of segmentation accuracy, while requiring only about $1\%$ of the parameters. Additionally, we observe that the architecture can be easily and effectively adapted to a new domain without sacrificing performance in the domains used to learn the shared parameterization of the universal network. We put the code of 3D U$^2$-Net into public domain. \url{https://github.com/huangmozhilv/u2net_torch/}

Figures

Figures reproduced from arXiv: 1909.06012 by the authors.

Figure 1
Figure 1. Domain adapter based on separable convolution. a a a a 16 a a a a a a a a 32 64 128 256 512 Down Sampling Up Sampling a a a a a a a a a a Residual Connection Concatenation + + 111 convolution a Adapter Multi-level segmentations [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The proposed 3D Universal U-Net (3D U2 -Net). applying C 0 filters w ∈ R 3×3×3×C on the input in parallel and concatenating the C 0 output feature maps. A simple calculation tells that the total number of filter parameters in the above filters is 27 ∗ C ∗ C 0 . Also, when training the models for the T domains separately, the number of parameters grows T times! In separable convolution, the computation is factorized … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 13 canonical work pages

  1. [1]

    In: Proc

    Berman, M., Rannen Triki, A., Blaschko, M.B.: The lov´ asz-softmax loss: a tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In: Proc. CVPR. pp. 4413–4421 (2018)

    work page 2018

  2. [2]

    arXiv:1701.07275 (2017)

    Bilen, H., Vedaldi, A.: Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv:1701.07275 (2017)

    arXiv 2017

  3. [3]

    In: Proc

    Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proc. CVPR. pp. 1251–1258 (2017)

    work page 2017

  4. [4]

    In: Proc

    C ¸ i¸ cek,¨O., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-net: learning dense volumetric segmentation from sparse annotation. In: Proc. MICCAI. pp. 424–432. Springer (2016)

    work page 2016

  5. [5]

    arXiv:1902.00927 (2019)

    Guo, Y., Li, Y., Feris, R., Wang, L., Rosing, T.: Depthwise convolution is all you need for learning multiple visual domains. arXiv:1902.00927 (2019)

    arXiv 2019

  6. [6]

    arXiv:1809.10486 (2018)

    Isensee, F., Petersen, J., Klein, A., Zimmerer, D., Jaeger, P.F., Kohl, S., Wasserthal, J., Koehler, G., Norajitra, T., Wirkert, S., et al.: nnU-Net: Self- adapting framework for u-net-based medical image segmentation. arXiv:1809.10486 (2018)

    arXiv 2018

  7. [7]

    In: Proc

    Karani, N., Chaitanya, K., Baumgartner, C., Konukoglu, E.: A lifelong learning approach to brain MR segmentation across scanners and protocols. In: Proc. MIC- CAI. pp. 476–484. Springer (2018)

    work page 2018

  8. [8]

    arXiv:1701.03056 (2017)

    Kayalibay, B., Jensen, G., van der Smagt, P.: CNN-based segmentation of medical imaging data. arXiv:1701.03056 (2017)

    arXiv 2017

Show all 19 references
  1. [9]

    In: Proc

    Lay, N., Birkbeck, N., Zhang, J., Zhou, S.K.: Rapid multi-organ segmentation using context integration and discriminative models. In: Proc. MICCAI. pp. 450–462. Springer (2013)

    2013

  2. [10]

    In: Proc

    Lin, T.Y., Goyal, P., Girshick, R., He, K., Doll´ ar, P.: Focal loss for dense object detection. In: Proc. ICCV. pp. 2980–2988 (2017) 3D U2-Net: A 3D Universal U-Net for Multi-Domain Segmentation 9

    2017

  3. [11]

    In: Proc

    Milletari, F., Navab, N., Ahmadi, S.A.: V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In: Proc. 3DV. pp. 565–571 (2016)

    2016

  4. [12]

    In: Proc

    Moeskops, P., Wolterink, J.M., van der Velden, B.H., Gilhuijs, K.G., Leiner, T., Viergever, M.A., Iˇ sgum, I.: Deep learning for multi-task medical image segmenta- tion in multiple modalities. In: Proc. MICCAI. pp. 478–486. Springer (2016)

    2016

  5. [13]

    In: Proc

    Rebuffi, S.A., Bilen, H., Vedaldi, A.: Learning multiple visual domains with residual adapters. In: Proc. NIPS. pp. 506–516 (2017)

    2017

  6. [14]

    In: Proc

    Rebuffi, S.A., Bilen, H., Vedaldi, A.: Efficient parametrization of multi-domain deep neural networks. In: Proc. CVPR. pp. 8119–8127 (2018)

    2018

  7. [15]

    In: Proc

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: Proc. MICCAI. pp. 234–241. Springer (2015)

    2015

  8. [16]

    In: Proc

    Roth, H.R., Lu, L., Farag, A., Shin, H.C., Liu, J., Turkbey, E.B., Summers, R.M.: DeepOrgan: Multi-level deep convolutional networks for automated pancreas seg- mentation. In: Proc. MICCAI. pp. 556–564. Springer (2015)

    2015

  9. [17]

    arXiv:1704.06382 (2017)

    Roth, H.R., Oda, H., Hayashi, Y., Oda, M., Shimizu, N., Fujiwara, M., Misawa, K., Mori, K.: Hierarchical 3D fully convolutional networks for multi-organ segmen- tation. arXiv:1704.06382 (2017)

    2017 arXiv

  10. [18]

    arXiv:1808.01944 (2018)

    Savioli, N., Montana, G., Lamata, P.: V-FCNN: Volumetric fully convolution neural network for automatic atrial segmentation. arXiv:1808.01944 (2018)

    2018 arXiv

  11. [19]

    arXiv:1902.09063 (2019)

    Simpson, A.L., Antonelli, M., Bakas, S., et al.: A large annotated medical im- age dataset for the development and evaluation of segmentation algorithms. arXiv:1902.09063 (2019)

    2019 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.