Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

MULAN: Multitask Universal Lesion Analysis Network for Joint Lesion Detection, Tagging, and Segmentation

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read MULAN joins detection, tagging, and segmentation of CT lesions in one end-to-end network, achieving 86.12% average sensitivity on DeepLesion.

desk verdict Solid empirical contribution to universal lesion analysis, but the claim that tag predictions improve detection is undercut by a confounded ablation; fix that and it's an accept. read the letter →

arxiv 1908.04373 v1 pith:I46RSCBP submitted 2019-08-12 cs.CV

classification cs.CV
keywords MULANuniversallesiondetectionDeepmultilabeltaggingMaskR-CNN3DfeaturefusionscorerefinementlayerCTimaging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MULAN is a single deep network that performs three tasks radiologists do when reading a CT scan: find lesions, characterize them with semantic tags (body part, type, attributes), and measure them by segmentation. The paper claims state-of-the-art results on the DeepLesion dataset: 86.12% average detection sensitivity versus 75.55% for the previous best method, and 96.01% mean tagging AUC versus 95.12% for the prior tagging baseline. The central causal claim is that tag predictions improve detection accuracy through a score refinement layer: removing that layer drops detection to 84.24%, while adding it raises detection to 86.12%. This matters because it shows that a single model can approximate a radiologist's full workflow and that semantically meaningful tag features can help localize lesions.

What carries the argument

The score refinement layer (SRL) is a single fully connected layer that takes the lesion score, 185 tag scores, proposal box statistics (x, y, width, height), and patient age and gender as input, and outputs refined lesion and tag scores. The other load-bearing component is 3D feature fusion, which inserts fusion layers after dense block 2 and after the feature pyramid to merge feature maps from three consecutive CT slices, replacing the central feature map with the 3D-context-fused version while keeping the neighboring maps unchanged. The tagging branch is trained on 185 tags mined from radiology reports using label expansion and relational hard example mining to handle incomplete and noisy text-derived labels.

What would settle it

Retrain MULAN with a control score refinement layer that receives the same box statistics and demographics plus tag scores whose values are randomly permuted across proposals, keeping everything else fixed. If the detection sensitivity with permuted tag scores matches the full MULAN result (86.12%) rather than dropping toward the no-refinement value (84.24%), the claim that semantic tag predictions drive the improvement is falsified.

Watch

Extended reading notes

Core claim

MULAN, an improved Mask R-CNN with three head branches and a 3D feature fusion strategy, achieves state-of-the-art universal lesion detection (86.12% average sensitivity, compared to 75.55% for 3DCE) and state-of-the-art lesion tagging (96.01% mean AUC, compared to 95.12% for LesaNet) on the DeepLesion dataset. The paper further establishes that the score refinement layer, which re-predicts lesion and tag scores from the concatenation of the detection score, tag scores, box statistics, and patient age/gender, is what makes tag predictions improve detection: ablating it lowers detection by nearly two points, and five random re-splits consistently show a gain of 0.7–1.1%.

Load-bearing premise

The mined tag labels from radiology reports are sufficiently accurate and complete to train the tagging branch and the score refinement layer; if these text-derived labels carry systematic noise that correlates with lesion confidence rather than true lesion attributes, the claimed detection gain from the score refinement layer could be an artifact.

Editorial extensions

If this is right

  • A single universal model can handle detection, tagging, and segmentation across body parts, extending prior work that focused on one organ or one task; this could streamline radiology workflows that currently require separate tools.
  • Semantic tag predictions measurably improve detection accuracy via score refinement, indicating that lesion attributes carry information useful for distinguishing true lesions from false positives.
  • The detection task itself improves tagging accuracy, while it impairs segmentation performance, suggesting that task relationships are asymmetric and need explicit handling in multitask designs.
  • Feature pyramid and 3D fusion contribute the largest gains to detection, so these components are likely valuable for other volumetric medical detection tasks.
  • The released code and the 171-tag variant allow the community to reproduce the results and train on the same tag vocabulary as related work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The score refinement layer's success implies that a richer tag vocabulary or structured report information could further boost detection, beyond the current 185 mined tags.
  • The observed interference between detection and segmentation suggests that adaptive loss weighting or gradient modulation might recover some segmentation accuracy without giving up the detection and tagging benefits.
  • The 3D feature fusion strategy is architecture-agnostic and could be transferred to other 2D detectors for volumetric medical images, potentially improving localization in MRI or ultrasound volumes as well.
  • Because the tag labels come from text mining, the apparent detection gain from SRL might partly reflect correlation between tag confidence and lesion confidence; a control experiment with permuted tag scores would clarify whether semantic content or mere confidence is driving the improvement.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The manuscript proposes MULAN, a Mask R-CNN-based multitask network for joint lesion detection, tagging, and segmentation in CT images. MULAN uses a DenseNet backbone with a feature pyramid and a 3D feature-fusion strategy that fuses feature maps of the central slice with adjacent slices at two network depths. The network has three head branches plus a score refinement layer (SRL) that takes the detection score, 185 mined tag scores, proposal box statistics, and patient gender and age as input, and outputs refined lesion and tag scores. On the official DeepLesion split, MULAN reports an average detection sensitivity of 86.12% (vs 75.55% for 3DCE), a tagging AUC of 96.01% (vs 95.12% for a rerun of LesaNet), and surrogate segmentation metrics of 1.41 mm endpoint distance and 1.97 mm diameter error. Ablations show positive contributions of the feature pyramid, 3D fusion, and the score refinement layer. The paper concludes that tag predictions improve detection accuracy via the SRL.

Significance. If the claims hold, this is a strong contribution to universal lesion analysis. The 3D feature fusion strategy is a sensible extension of 3DCE and yields a large detection improvement. The study uses the official data split, compares against previous published numbers and a rerun of LesaNet, includes thorough ablations, and provides five random re-splits to verify the SRL effect. The authors have released the code. The main weakness is that the specific claim that tag predictions improve detection is not isolated by the reported ablations, because the SRL also consumes non-tag features; an additional ablation is needed to support the stated mechanism.

major comments (2)
  1. [Section 3, Table 1 (ablation f); Section 2.2] The claim that tag predictions improve detection accuracy is underdetermined. In Section 2.2, the SRL is described as joining the lesion score, the 185 tag scores, and 'new features' including the proposal statistics (x, y, width, height), patient gender, and age. Ablation (f) removes the entire SRL, so the 1.88-point gain (86.12% vs 84.24%) conflates the contribution of tag scores with box geometry and demographic features, both of which plausibly correlate with lesion likelihood. The five random re-splits compare full MULAN with SRL against MULAN without SRL and therefore inherit the same confound. An ablation that keeps the SRL and all non-tag input features while removing only the 185 tag scores is necessary to support the paper's central conclusion that tag predictions improve detection.
  2. [Section 3, Table 1] The main detection results are reported on a single official split without error bars or multiple runs. The five random re-splits reported for the SRL comparison do not quantify the variance of the headline detection comparison against 3DCE and ULDor. Since the SRL gain (1.88 points) is much smaller than the gain over 3DCE (10.57 points), reporting variance or at least clarifying the number of runs for the main results would strengthen the empirical claims.
minor comments (4)
  1. [Section 3] The text says MULAN outperforms previous work 'by over 10%' in average sensitivity; since the comparison is 86.12% vs 75.55%, this should read 'by over 10 percentage points' to avoid ambiguity.
  2. [Appendix 5.1] There is a typo: 'psuedo-mask' should be 'pseudo-mask'.
  3. [Section 2.2 and Section 5.2] The description that the score refinement layer was 'initialized with an identity matrix' is dimensionally ambiguous: the input dimension (1 lesion score + 185 tag scores + 6 additional features) is not equal to the output dimension (1 refined lesion score + 185 refined tag scores). Please clarify how the identity initialization is applied, for example by initializing the submatrix corresponding to the lesion and tag scores to identity and the rest to zero.
  4. [Table 1 caption] The segmentation metrics 'Distance' and 'Diam. err.' are given without units in the table caption; the text states 'mm' in Section 3, but the caption should also include units.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MULAN's results are produced by standard train/test evaluation, and its self-citations are to external datasets and baselines rather than to the paper's own outputs.

full rationale

The paper's central derivations are empirical: MULAN is trained on the official DeepLesion split and evaluated on held-out test data, with detection, tagging, and segmentation metrics all computed on independent ground truth. The tag labels are mined from radiological reports and evaluated on 500 manually tagged lesions from prior work [16], not on MULAN's own predictions. The score refinement layer (SRL) is trained end-to-end and its effect is tested by removing it, with the improvement replicated across five random re-splits of the training/validation data. Self-citations in the paper are to the DeepLesion dataset, the 3DCE detector, ULDor, LesaNet, and Auto RECIST; these are prior external resources and baselines, and none of them is used to define MULAN's outputs or to justify a result by construction. The SRL ablation removes tag scores together with proposal statistics, gender, and age, so the claim that tag predictions specifically drive the detection improvement is not fully isolated experimentally; however, this is a methodological confound, not circularity, since no equation or definition makes the detection result logically equivalent to the tag inputs. No step was found where a fitted parameter is renamed as a prediction, a uniqueness theorem is imported from the authors' own prior work, or a known result is repackaged under new coordinates.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

MULAN is an empirical deep-learning system; the central claim is a measured performance on a benchmark, not a mathematical derivation. The listed free parameters are hand-set hyperparameters and preprocessing choices. The axioms are the data and label assumptions on which the model relies, including the pseudo-mask proxy for segmentation ground truth and the validity of text-mined tags. No invented physical entities are introduced.

free parameters (6)
  • Bounding-box regression loss weight = 10
    Hand-set weighting in the combined loss (Eq. 1); no ablation is reported.
  • 3D fusion layer placement = after dense block 2 and after last feature pyramid layer
    Chosen manually; fusing earlier was observed to be worse but the comparison is not shown.
  • Tag frequency threshold = >30 occurrences
    Cutoff selects the final 185 tags; changing this threshold changes the label set.
  • Input sub-volume depth and z-resampling = 9 slices, resampled to 2mm spacing
    Preprocessing choice that defines the amount of 3D context available to the fusion layers.
  • Anchor geometry = scales 16,24,32,48,96; ratios 1:2,1:1,2:1
    Hand-set RPN anchors, not optimized.
  • Learning rate schedule = base 0.004, decay by 10 at epochs 4 and 6, 8 epochs
    Standard SGD schedule chosen by hand.
assumptions (5)
  • domain assumption Pseudo-masks built from RECIST quadrant approximations are a valid proxy for true lesion masks when training the segmentation branch.
    DeepLesion has no ground-truth masks; the paper generates four quadrants from the annotated long and short axes (Section 2.2).
  • domain assumption Text-mined tags, after label expansion and relational hard example mining, are accurate enough to serve as ground truth for the tagging branch and the score refinement layer.
    Tag mining from radiology reports and the RHEM strategy are described in Sections 2.2 and 5.2; only 500 lesions are manually tagged for evaluation.
  • domain assumption Surrogate metrics (distance from GT RECIST endpoints to predicted contour and RECIST diameter error) reflect clinically meaningful segmentation quality.
    Defined in Section 5.3 because no true masks exist; their clinical validity is assumed.
  • domain assumption The single windowing, 0.8mm in-plane resampling, and 2mm z-interpolation preserve lesion information needed for detection, tagging, and segmentation.
    Preprocessing described in Section 5.3 with no comparative study of alternative settings.
  • ad hoc to paper Fusing feature maps of the central slice with adjacent slices at two network depths yields beneficial 3D context without harming localization.
    Proposed architectural assumption in Section 2.1, supported only by the ablation in Table 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MULAN: Multitask Universal Lesion Analysis Network for Joint Lesion Detection, Tagging, and Segmentation." pith.science (2026). https://pith.science/paper/I46RSCBP

@misc{pith2026190804373,
  author       = {Pith},
  title        = {Pith review of: MULAN: Multitask Universal Lesion Analysis Network for Joint Lesion Detection, Tagging, and Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I46RSCBP}},
  note         = {Machine review of arXiv:1908.04373}
}
read the original abstract

When reading medical images such as a computed tomography (CT) scan, radiologists generally search across the image to find lesions, characterize and measure them, and then describe them in the radiological report. To automate this process, we propose a multitask universal lesion analysis network (MULAN) for joint detection, tagging, and segmentation of lesions in a variety of body parts, which greatly extends existing work of single-task lesion analysis on specific body parts. MULAN is based on an improved Mask R-CNN framework with three head branches and a 3D feature fusion strategy. It achieves the state-of-the-art accuracy in the detection and tagging tasks on the DeepLesion dataset, which contains 32K lesions in the whole body. We also analyze the relationship between the three tasks and show that tag predictions can improve detection accuracy via a score refinement layer.

Figures

Figures reproduced from arXiv: 1908.04373 by the authors.

Figure 1
Figure 1. Flowchart of MULAN and the 3D feature fusion strategy. the backbone with the last dense block and transition layer removed, as we found removing them slightly improved accuracy and speed. Next, we employ the feature pyramid strategy [7] to add fine-level details into the feature map. This strategy also increases the size of the final feature map, which will benefit the detection and segmentation of small lesions. Di… view at source ↗
Figure 2
Figure 2. Illustration of the head branches and the score refinement layer of MULAN. The structure and function of the three head branches are shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples of MULAN’s lesion detection, tagging, and segmentation results on the test set of DeepLesion. For detection, boxes in green and red are predicted TPs and FPs, respectively. The number above each box is the lesion score (confidence). For tagging, tags in black, red (underlined), and blue (italic) are predicted TPs, FPs, and FNs, respectively. They are ranked by their scores. For segmentation, the green lines… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Examples of the CT images, annotations, and reports in DeepLesion [17]. The red and blue lines in the images are the RECIST measurements. The green boxes are the bounding-boxes. The sentences are extracted from radiological reports according to the bookmarks [16]. The …
Figure 5
Figure 5. Figure 5: Illustration of the predicted mask (green contour), estimated RECIST mea￾surement (green segments), and ground-truth RECIST measurement (orange segments with yellow endpoints) of a lesion. Surrogate Evaluation Criteria for Lesion Segmentation There are no ground-truth …
Figure 6
Figure 6. Figure 6: Examples of MULAN’s lesion detection, tagging, and segmentation results on the test set of DeepLesion. For detection, boxes in green and red are predicted TPs and FPs, respectively. The number above each box is the lesion score (confidence). For tagging, tags in black,…
Figure 7
Figure 7. Figure 7: Free-response receiver operating characteristic (FROC) curve of various meth￾ods and variations of MULAN on the test set of DeepLesion [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TUNA-Net: Task-oriented UNsupervised Adversarial Network for Disease Recognition in Cross-Domain Chest X-rays

    eess.IV 2019-08 conditional novelty 6.0 of 10

    TUNA-Net, a task-driven cycle-consistent GAN, achieves 96.3% AUC for pediatric pneumonia detection without using pediatric labels by preserving disease semantics during adult-to-pediatric X-ray translation.

Reference graph

Works this paper leans on

17 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Yan et al

    Diamant, I., Hoogi, A., Beaulieu, C.F., Safdari, M., Klang, E., Amitai, M., Greenspan, H., Rubin, D.L.: Improved Patch-Based Automated Liver Lesion Clas- 3 https://github.com/rsummers11/CADLab/tree/master/LesaNet 14 K. Yan et al. lung mass, right upper lobe, perihilar, right lower lobe, nodule, lobular, spiculated right hilum lymph node, perihilar pancrea...

  2. [2]

    European Journal of Cancer 45(2), 228–247 (2009)

    Eisenhauer, E.A., Therasse, P., Bogaerts, J., Schwartz, L.H., Sargent, D., Ford, R., Dancey, J., Arbuck, S., Gwyther, S., Mooney, M., Rubinstein, L., Shankar, L., Dodd, L., Kaplan, R., Lacombe, D., Verweij, J.: New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1). European Journal of Cancer 45(2), 228–247 (2009)

  3. [3]

    In: ICCV

    He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask R-CNN. In: ICCV. pp. 2980– 2988 (2017)

  4. [4]

    In: CVPR (2017)

    Huang, G., Liu, Z., Weinberger, K.Q., van der Maaten, L.: Densely Connected Convolutional Networks. In: CVPR (2017)

  5. [5]

    Radiographics 26(6), 1595–1597 (Nov 2006)

    Langlotz, C.P.: RadLex: a new method for indexing online educational materials. Radiographics 26(6), 1595–1597 (Nov 2006)

  6. [6]

    IEEE Transactions on Neural Networks and Learning Systems (2019)

    Liao, F., Liang, M., Li, Z., Hu, X., Song, S.: Evaluate the Malignancy of Pulmonary Nodules Using the 3D Deep Leaky Noisy-or Network. IEEE Transactions on Neural Networks and Learning Systems (2019)

  7. [7]

    In: CVPR (2017)

    Lin, T.Y., Doll´ ar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR (2017)

  8. [8]

    In: International Conference on 3D Vision

    Milletari, F., Navab, N., Ahmadi, S.A.: V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In: International Conference on 3D Vision. pp. 565–571 (2016)

Show all 17 references
  1. [9]

    Scientific Reports 8(1) (2018)

    Ribli, D., Horv´ ath, A., Unger, Z., Pollner, P., Csabai, I.: Detecting and classifying lesions in mammograms with Deep Learning. Scientific Reports 8(1) (2018)

  2. [10]

    Medical Physics (oct 2018)

    Sahiner, B., Pezeshk, A., Hadjiiski, L.M., Wang, X., Drukker, K., Cha, K.H., Sum- mers, R.M., Giger, M.L.: Deep learning in medical imaging and radiation therapy. Medical Physics (oct 2018)

  3. [11]

    In: MICCAI (jun 2018), http://arxiv.org/abs/1806.09507

    Tang, Y., Harrison, A.P., Bagheri, M., Xiao, J., Summers, R.M.: Semi-Automatic RECIST Labeling on CT Scans with Cascaded Convolutional Neural Networks. In: MICCAI (jun 2018), http://arxiv.org/abs/1806.09507

  4. [12]

    In: SPIE

    Tang, Y., Oh, S., Xiao, J., Summers, R.M., Tang, Y.: CT-realistic data augmen- tation using generative adversarial network for robust lymph node segmentation. In: SPIE. p. 109503V (2019). https://doi.org/10.1117/12.2512004

  5. [13]

    In: ISBI (2019)

    Tang, Y., Yan, K., Tang, Y.X., Liu, J., Xiao, J., Summers, R.M.: ULDor: A Univer- sal Lesion Detector for CT Scans with Pseudo Masks and Hard Negative Example Mining. In: ISBI (2019)

  6. [14]

    In: ISBI

    Wu, B., Zhou, Z., Wang, J., Wang, Y.: Joint learning for pulmonary nodule seg- mentation, attributes and malignancy prediction. In: ISBI. pp. 1109–1113 (2018)

  7. [15]

    In: MICCAI

    Yan, K., Bagheri, M., Summers, R.M.: 3D Context Enhanced Region-Based Con- volutional Neural Network for End-to-End Lesion Detection. In: MICCAI. pp. 511–519 (2018)

  8. [16]

    In: CVPR (2019)

    Yan, K., Peng, Y., Sandfort, V., Bagheri, M., Lu, Z., Summers, R.M.: Holistic and Comprehensive Annotation of Clinically Significant Findings on Diverse CT Images : Learning from Radiology Reports and Label Ontology. In: CVPR (2019)

  9. [17]

    Journal of Medical Imaging 5(3) (2018)

    Yan, K., Wang, X., Lu, L., Summers, R.M.: DeepLesion: automated mining of large- scale lesion annotations and universal lesion detection with deep learning. Journal of Medical Imaging 5(3) (2018). https://doi.org/10.1117/1.JMI.5.3.036501

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.