REVIEW 2 major objections 4 minor 1 cited by
MULAN: Multitask Universal Lesion Analysis Network for Joint Lesion Detection, Tagging, and Segmentation
T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read MULAN joins detection, tagging, and segmentation of CT lesions in one end-to-end network, achieving 86.12% average sensitivity on DeepLesion.
desk verdict Solid empirical contribution to universal lesion analysis, but the claim that tag predictions improve detection is undercut by a confounded ablation; fix that and it's an accept. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The score refinement layer (SRL) is a single fully connected layer that takes the lesion score, 185 tag scores, proposal box statistics (x, y, width, height), and patient age and gender as input, and outputs refined lesion and tag scores. The other load-bearing component is 3D feature fusion, which inserts fusion layers after dense block 2 and after the feature pyramid to merge feature maps from three consecutive CT slices, replacing the central feature map with the 3D-context-fused version while keeping the neighboring maps unchanged. The tagging branch is trained on 185 tags mined from radiology reports using label expansion and relational hard example mining to handle incomplete and noisy text-derived labels.
What would settle it
Retrain MULAN with a control score refinement layer that receives the same box statistics and demographics plus tag scores whose values are randomly permuted across proposals, keeping everything else fixed. If the detection sensitivity with permuted tag scores matches the full MULAN result (86.12%) rather than dropping toward the no-refinement value (84.24%), the claim that semantic tag predictions drive the improvement is falsified.
Extended reading notes
Core claim
MULAN, an improved Mask R-CNN with three head branches and a 3D feature fusion strategy, achieves state-of-the-art universal lesion detection (86.12% average sensitivity, compared to 75.55% for 3DCE) and state-of-the-art lesion tagging (96.01% mean AUC, compared to 95.12% for LesaNet) on the DeepLesion dataset. The paper further establishes that the score refinement layer, which re-predicts lesion and tag scores from the concatenation of the detection score, tag scores, box statistics, and patient age/gender, is what makes tag predictions improve detection: ablating it lowers detection by nearly two points, and five random re-splits consistently show a gain of 0.7–1.1%.
Load-bearing premise
The mined tag labels from radiology reports are sufficiently accurate and complete to train the tagging branch and the score refinement layer; if these text-derived labels carry systematic noise that correlates with lesion confidence rather than true lesion attributes, the claimed detection gain from the score refinement layer could be an artifact.
Editorial extensions
If this is right
- A single universal model can handle detection, tagging, and segmentation across body parts, extending prior work that focused on one organ or one task; this could streamline radiology workflows that currently require separate tools.
- Semantic tag predictions measurably improve detection accuracy via score refinement, indicating that lesion attributes carry information useful for distinguishing true lesions from false positives.
- The detection task itself improves tagging accuracy, while it impairs segmentation performance, suggesting that task relationships are asymmetric and need explicit handling in multitask designs.
- Feature pyramid and 3D fusion contribute the largest gains to detection, so these components are likely valuable for other volumetric medical detection tasks.
- The released code and the 171-tag variant allow the community to reproduce the results and train on the same tag vocabulary as related work.
Reading between the lines
- The score refinement layer's success implies that a richer tag vocabulary or structured report information could further boost detection, beyond the current 185 mined tags.
- The observed interference between detection and segmentation suggests that adaptive loss weighting or gradient modulation might recover some segmentation accuracy without giving up the detection and tagging benefits.
- The 3D feature fusion strategy is architecture-agnostic and could be transferred to other 2D detectors for volumetric medical images, potentially improving localization in MRI or ultrasound volumes as well.
- Because the tag labels come from text mining, the apparent detection gain from SRL might partly reflect correlation between tag confidence and lesion confidence; a control experiment with permuted tag scores would clarify whether semantic content or mere confidence is driving the improvement.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes MULAN, a Mask R-CNN-based multitask network for joint lesion detection, tagging, and segmentation in CT images. MULAN uses a DenseNet backbone with a feature pyramid and a 3D feature-fusion strategy that fuses feature maps of the central slice with adjacent slices at two network depths. The network has three head branches plus a score refinement layer (SRL) that takes the detection score, 185 mined tag scores, proposal box statistics, and patient gender and age as input, and outputs refined lesion and tag scores. On the official DeepLesion split, MULAN reports an average detection sensitivity of 86.12% (vs 75.55% for 3DCE), a tagging AUC of 96.01% (vs 95.12% for a rerun of LesaNet), and surrogate segmentation metrics of 1.41 mm endpoint distance and 1.97 mm diameter error. Ablations show positive contributions of the feature pyramid, 3D fusion, and the score refinement layer. The paper concludes that tag predictions improve detection accuracy via the SRL.
Significance. If the claims hold, this is a strong contribution to universal lesion analysis. The 3D feature fusion strategy is a sensible extension of 3DCE and yields a large detection improvement. The study uses the official data split, compares against previous published numbers and a rerun of LesaNet, includes thorough ablations, and provides five random re-splits to verify the SRL effect. The authors have released the code. The main weakness is that the specific claim that tag predictions improve detection is not isolated by the reported ablations, because the SRL also consumes non-tag features; an additional ablation is needed to support the stated mechanism.
major comments (2)
- [Section 3, Table 1 (ablation f); Section 2.2] The claim that tag predictions improve detection accuracy is underdetermined. In Section 2.2, the SRL is described as joining the lesion score, the 185 tag scores, and 'new features' including the proposal statistics (x, y, width, height), patient gender, and age. Ablation (f) removes the entire SRL, so the 1.88-point gain (86.12% vs 84.24%) conflates the contribution of tag scores with box geometry and demographic features, both of which plausibly correlate with lesion likelihood. The five random re-splits compare full MULAN with SRL against MULAN without SRL and therefore inherit the same confound. An ablation that keeps the SRL and all non-tag input features while removing only the 185 tag scores is necessary to support the paper's central conclusion that tag predictions improve detection.
- [Section 3, Table 1] The main detection results are reported on a single official split without error bars or multiple runs. The five random re-splits reported for the SRL comparison do not quantify the variance of the headline detection comparison against 3DCE and ULDor. Since the SRL gain (1.88 points) is much smaller than the gain over 3DCE (10.57 points), reporting variance or at least clarifying the number of runs for the main results would strengthen the empirical claims.
minor comments (4)
- [Section 3] The text says MULAN outperforms previous work 'by over 10%' in average sensitivity; since the comparison is 86.12% vs 75.55%, this should read 'by over 10 percentage points' to avoid ambiguity.
- [Appendix 5.1] There is a typo: 'psuedo-mask' should be 'pseudo-mask'.
- [Section 2.2 and Section 5.2] The description that the score refinement layer was 'initialized with an identity matrix' is dimensionally ambiguous: the input dimension (1 lesion score + 185 tag scores + 6 additional features) is not equal to the output dimension (1 refined lesion score + 185 refined tag scores). Please clarify how the identity initialization is applied, for example by initializing the submatrix corresponding to the lesion and tag scores to identity and the rest to zero.
- [Table 1 caption] The segmentation metrics 'Distance' and 'Diam. err.' are given without units in the table caption; the text states 'mm' in Section 3, but the caption should also include units.
Circularity Check
No significant circularity: MULAN's results are produced by standard train/test evaluation, and its self-citations are to external datasets and baselines rather than to the paper's own outputs.
full rationale
The paper's central derivations are empirical: MULAN is trained on the official DeepLesion split and evaluated on held-out test data, with detection, tagging, and segmentation metrics all computed on independent ground truth. The tag labels are mined from radiological reports and evaluated on 500 manually tagged lesions from prior work [16], not on MULAN's own predictions. The score refinement layer (SRL) is trained end-to-end and its effect is tested by removing it, with the improvement replicated across five random re-splits of the training/validation data. Self-citations in the paper are to the DeepLesion dataset, the 3DCE detector, ULDor, LesaNet, and Auto RECIST; these are prior external resources and baselines, and none of them is used to define MULAN's outputs or to justify a result by construction. The SRL ablation removes tag scores together with proposal statistics, gender, and age, so the claim that tag predictions specifically drive the detection improvement is not fully isolated experimentally; however, this is a methodological confound, not circularity, since no equation or definition makes the detection result logically equivalent to the tag inputs. No step was found where a fitted parameter is renamed as a prediction, a uniqueness theorem is imported from the authors' own prior work, or a known result is repackaged under new coordinates.
Assumptions & free parameters
free parameters (6)
- Bounding-box regression loss weight =
10
- 3D fusion layer placement =
after dense block 2 and after last feature pyramid layer
- Tag frequency threshold =
>30 occurrences
- Input sub-volume depth and z-resampling =
9 slices, resampled to 2mm spacing
- Anchor geometry =
scales 16,24,32,48,96; ratios 1:2,1:1,2:1
- Learning rate schedule =
base 0.004, decay by 10 at epochs 4 and 6, 8 epochs
assumptions (5)
- domain assumption Pseudo-masks built from RECIST quadrant approximations are a valid proxy for true lesion masks when training the segmentation branch.
- domain assumption Text-mined tags, after label expansion and relational hard example mining, are accurate enough to serve as ground truth for the tagging branch and the score refinement layer.
- domain assumption Surrogate metrics (distance from GT RECIST endpoints to predicted contour and RECIST diameter error) reflect clinically meaningful segmentation quality.
- domain assumption The single windowing, 0.8mm in-plane resampling, and 2mm z-interpolation preserve lesion information needed for detection, tagging, and segmentation.
- ad hoc to paper Fusing feature maps of the central slice with adjacent slices at two network depths yields beneficial 3D context without harming localization.
Cite this review
Pith. "Pith review of MULAN: Multitask Universal Lesion Analysis Network for Joint Lesion Detection, Tagging, and Segmentation." pith.science (2026). https://pith.science/paper/I46RSCBP
@misc{pith2026190804373,
author = {Pith},
title = {Pith review of: MULAN: Multitask Universal Lesion Analysis Network for Joint Lesion Detection, Tagging, and Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/I46RSCBP}},
note = {Machine review of arXiv:1908.04373}
}
read the original abstract
When reading medical images such as a computed tomography (CT) scan, radiologists generally search across the image to find lesions, characterize and measure them, and then describe them in the radiological report. To automate this process, we propose a multitask universal lesion analysis network (MULAN) for joint detection, tagging, and segmentation of lesions in a variety of body parts, which greatly extends existing work of single-task lesion analysis on specific body parts. MULAN is based on an improved Mask R-CNN framework with three head branches and a 3D feature fusion strategy. It achieves the state-of-the-art accuracy in the detection and tagging tasks on the DeepLesion dataset, which contains 32K lesions in the whole body. We also analyze the relationship between the three tasks and show that tag predictions can improve detection accuracy via a score refinement layer.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
TUNA-Net: Task-oriented UNsupervised Adversarial Network for Disease Recognition in Cross-Domain Chest X-rays
TUNA-Net, a task-driven cycle-consistent GAN, achieves 96.3% AUC for pediatric pneumonia detection without using pediatric labels by preserving disease semantics during adult-to-pediatric X-ray translation.
Reference graph
Works this paper leans on
-
[1]
Diamant, I., Hoogi, A., Beaulieu, C.F., Safdari, M., Klang, E., Amitai, M., Greenspan, H., Rubin, D.L.: Improved Patch-Based Automated Liver Lesion Clas- 3 https://github.com/rsummers11/CADLab/tree/master/LesaNet 14 K. Yan et al. lung mass, right upper lobe, perihilar, right lower lobe, nodule, lobular, spiculated right hilum lymph node, perihilar pancrea...
work page 2016
-
[2]
European Journal of Cancer 45(2), 228–247 (2009)
Eisenhauer, E.A., Therasse, P., Bogaerts, J., Schwartz, L.H., Sargent, D., Ford, R., Dancey, J., Arbuck, S., Gwyther, S., Mooney, M., Rubinstein, L., Shankar, L., Dodd, L., Kaplan, R., Lacombe, D., Verweij, J.: New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1). European Journal of Cancer 45(2), 228–247 (2009)
work page 2009
- [3]
-
[4]
Huang, G., Liu, Z., Weinberger, K.Q., van der Maaten, L.: Densely Connected Convolutional Networks. In: CVPR (2017)
work page 2017
-
[5]
Radiographics 26(6), 1595–1597 (Nov 2006)
Langlotz, C.P.: RadLex: a new method for indexing online educational materials. Radiographics 26(6), 1595–1597 (Nov 2006)
work page 2006
-
[6]
IEEE Transactions on Neural Networks and Learning Systems (2019)
Liao, F., Liang, M., Li, Z., Hu, X., Song, S.: Evaluate the Malignancy of Pulmonary Nodules Using the 3D Deep Leaky Noisy-or Network. IEEE Transactions on Neural Networks and Learning Systems (2019)
work page 2019
-
[7]
Lin, T.Y., Doll´ ar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR (2017)
work page 2017
-
[8]
In: International Conference on 3D Vision
Milletari, F., Navab, N., Ahmadi, S.A.: V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In: International Conference on 3D Vision. pp. 565–571 (2016)
work page 2016
Show all 17 references
-
[9]
Scientific Reports 8(1) (2018)
Ribli, D., Horv´ ath, A., Unger, Z., Pollner, P., Csabai, I.: Detecting and classifying lesions in mammograms with Deep Learning. Scientific Reports 8(1) (2018)
2018
-
[10]
Medical Physics (oct 2018)
Sahiner, B., Pezeshk, A., Hadjiiski, L.M., Wang, X., Drukker, K., Cha, K.H., Sum- mers, R.M., Giger, M.L.: Deep learning in medical imaging and radiation therapy. Medical Physics (oct 2018)
2018
-
[11]
In: MICCAI (jun 2018), http://arxiv.org/abs/1806.09507
Tang, Y., Harrison, A.P., Bagheri, M., Xiao, J., Summers, R.M.: Semi-Automatic RECIST Labeling on CT Scans with Cascaded Convolutional Neural Networks. In: MICCAI (jun 2018), http://arxiv.org/abs/1806.09507
2018 arXiv
-
[12]
In: SPIE
Tang, Y., Oh, S., Xiao, J., Summers, R.M., Tang, Y.: CT-realistic data augmen- tation using generative adversarial network for robust lymph node segmentation. In: SPIE. p. 109503V (2019). https://doi.org/10.1117/12.2512004
2019 doi
-
[13]
In: ISBI (2019)
Tang, Y., Yan, K., Tang, Y.X., Liu, J., Xiao, J., Summers, R.M.: ULDor: A Univer- sal Lesion Detector for CT Scans with Pseudo Masks and Hard Negative Example Mining. In: ISBI (2019)
2019
-
[14]
In: ISBI
Wu, B., Zhou, Z., Wang, J., Wang, Y.: Joint learning for pulmonary nodule seg- mentation, attributes and malignancy prediction. In: ISBI. pp. 1109–1113 (2018)
2018
-
[15]
In: MICCAI
Yan, K., Bagheri, M., Summers, R.M.: 3D Context Enhanced Region-Based Con- volutional Neural Network for End-to-End Lesion Detection. In: MICCAI. pp. 511–519 (2018)
2018
-
[16]
In: CVPR (2019)
Yan, K., Peng, Y., Sandfort, V., Bagheri, M., Lu, Z., Summers, R.M.: Holistic and Comprehensive Annotation of Clinically Significant Findings on Diverse CT Images : Learning from Radiology Reports and Label Ontology. In: CVPR (2019)
2019
-
[17]
Journal of Medical Imaging 5(3) (2018)
Yan, K., Wang, X., Lu, L., Summers, R.M.: DeepLesion: automated mining of large- scale lesion annotations and universal lesion detection with deep learning. Journal of Medical Imaging 5(3) (2018). https://doi.org/10.1117/1.JMI.5.3.036501
2018 doi
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.