FusionNet: Incorporating Shape and Texture for Abnormality Detection in 3D Abdominal CT Scans

Alan Yuille; Elliot Fishman; Fengze Liu; Yuyin Zhou

REVIEW 3 major objections 7 minor 17 references

FusionNet: Incorporating Shape and Texture for Abnormality Detection in 3D Abdominal CT Scans

T0 review · 3 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A network that multiplies pancreas-shape features with CT-texture features at its third layer detects pancreatic ductal adenocarcinoma with 92.65% sensitivity and 97.00% specificity.

desk verdict A useful applied paper whose headline gain is likely inflated by architecture selection on the same folds; worth reading for the fusion framework, but treat the 92/97 numbers as preliminary. read the letter →

arxiv 1908.07654 v2 pith:ZCZQ352F submitted 2019-08-21 cs.CV

Fengze Liu , Yuyin Zhou , Elliot Fishman , Alan Yuille This is my paper · ORCID

classification cs.CV

keywords pancreaticductaladenocarcinomaabnormalitydetection3DCTscansshapeandtexturefusionNetsegmentationmaskclassificationarchitecturesearch

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that pancreatic ductal adenocarcinoma can be detected in abdominal CT scans by combining two signals a tumor leaves behind: the deformed shape of the pancreas and the changed texture of the tissue. Rather than choosing one cue, the authors train a two-stage system that first segments the pancreas into a binary mask and then feeds that mask together with the CT image into a classification network. They search over fusion strategies and find the best is pointwise multiplication of the two branches' feature maps at the third convolutional layer. On 136 abnormal and 200 normal scans this FusionNet reaches 92.65% sensitivity and 97.00% specificity, improving by more than 10 sensitivity points and 5 specificity points over the image-only baseline. If this holds, a practical screening tool for pancreatic cancer is possible without lesion-level annotations.

What carries the argument

The load-bearing object is the parameterized fusion function $f_{\alpha\beta}(S,X;w)=f_{\alpha:L}(\beta(f_{1:\alpha}(S;w^1_{1:\alpha}), f_{1:\alpha}(X;w^2_{1:\alpha})); w_{\alpha:L})$, where $S$ is the predicted pancreas mask, $X$ is the CT volume, $\beta \in \{+,\ast,\oplus\}$, and $\alpha$ selects the layer where the two branches meet. Varying $\alpha$ and $\beta$ changes when and how shape and texture features are combined while keeping the post-fusion network fixed, so the architecture search isolates the fusion strategy. The winning configuration, multiplication at the third of six layers, is what the paper credits for the large jump over image-only and mask-only baselines.

What would settle it

Run the four-fold classification experiment with ground-truth pancreas masks, or with masks from a segmenter trained on abnormal scans, and check whether the more-than-10-point sensitivity gain over the image-only baseline persists; if it collapses or reverses, the reported complementarity depends on the specific segmentation model rather than on a general shape-texture fusion effect.

Watch

Extended reading notes

Core claim

The central claim is that shape and texture are complementary evidence for PDAC, and that explicit fusion in a deep network captures this complementarity better than either branch alone or naive score averaging. The paper defines a functional space of fusion architectures $f_{\alpha\beta}$ in which $\alpha$ is the layer at which the mask and image branches are combined and $\beta$ is the combination operation (addition, multiplication, or concatenation). A grid search over the 18 architectures shows the best model fuses at layer 3 with multiplication: FusionNet3*. This model reports 92.65% sensitivity and 97.00% specificity, while the image-only baseline achieves 83.09% sensitivity and 92.00% specificity. The authors interpret the large gap between naive fusion and FusionNet3* as evidence that the fusion architecture, not the mere availability of two inputs, carries the improvement.

Load-bearing premise

The predicted pancreas mask, produced by a segmentation network trained only on normal cases, stays a reliable shape descriptor when the same network is applied to scans that contain tumors.

Editorial extensions

If this is right

The two-stage pipeline can detect PDAC using only image-level labels for the classification stage and pancreas annotations only on normal scans, bypassing the need for lesion annotations.
Because the best fusion multiplies two feature streams mid-network, the same functional-space search can be transferred to other two-input classification tasks.
At 92.65% sensitivity and 97.00% specificity, the system misses 11 of 136 cancer cases and raises 6 false alarms per 200 normals, suggesting it could serve as a triage tool.
The reported more-than-10-point sensitivity gain over image-only input is the direct, quantitative consequence of fusing shape and texture.
The complementarity of mask and image implies that improvements in pancreas segmentation should translate directly into better PDAC detection within this framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the authors leave implicit: the same two-branch fusion recipe could apply to other cancers that visibly deform an organ, with the organ's segmentation mask serving as the shape channel.
A testable extension outside the paper: retrain the classification stage with masks from a segmentation network that has seen abnormal cases; if the gain persists, the complementarity is about shape and texture genuinely, not about mask errors.
The optimal fusion layer (third of six) may depend on dataset resolution and tumor size; on higher-resolution volumes the best $\alpha$ could shift, so a fixed architecture should be re-searched rather than assumed.
The paper compares against its own image-only and mask-only baselines; a fair clinical deployment would need external validation against radiologist reads and multi-center CT protocols.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing to know: this is an applied PDAC-detection paper that reports a large gain from fusing a predicted pancreas mask with the CT image, but the gain is probably inflated because the best of 18 fusion architectures was chosen on the same CV folds used to report final metrics. The strongest part is not the headline but the setup: image-level labels plus normal-case masks for segmentation, and a simple grid search over fusion layer/operation for a 3D VNet-style classifier. The oracle "Image+Mask GT" experiment, where either branch correct counts as correct, is a nice way to show complementarity, and it does support the claim that shape and texture carry different signal.

What is actually new: previous PDAC work used either mask (via autoencoder) or image; this paper shows on their dataset that fusing the two with a mid-level multiplicative connection outperforms both single branches and naive averaging. That is a legitimate empirical finding, not a new theory. The writing is clear and the comparison against AE+Mask is a reasonable baseline from the same group.

Soft spots, in order of size: 1. Selection bias. They grid search over 18 architectures in Section 3.1 and then report FusionNet3* — the best in Table 2 — as the method, with no held-out validation. With 34 abnormal cases per fold, a 9.6-point sensitivity gain is about 13 cases; picking the max of 18 correlated runs can easily produce that. This does not kill the paper, but it means the 92/97 numbers are optimistic, and the confidence interval around them is wide. A nested or separate validation split would fix this, and the paper would be much stronger for it. 2. No error bars or per-fold results, and the test set is a single private dataset. Minor-moderate. 3. Small inconsistency: abstract says 136 PDAC cases, conclusion says 138. That suggests sloppiness, but it doesn't change the logic. 4. The mask-reliability assumption: the segmentation network sees only normal cases, so on tumor cases the mask may be corrupted. The stress-test is right that this is secondary — if the mask is systematically off for PDAC cases, the shape branch can still carry signal, and the oracle analysis shows there is complementary information even if it is not pure "shape." Still, it would be worth an ablation with GT masks from a few abnormal cases if available.

Overall the central argument — that fusing mask and CT helps — holds up qualitatively, but the size of the improvement should be treated as preliminary. I would bring it to a reading group as an example of a practical two-stage medical imaging pipeline and of how easy it is to leak architecture selection into evaluation. It deserves serious peer review because the methodology is clear, the problem matters, and the flaw is fixable; a good referee would ask for nested CV and per-fold numbers.

Give it a conditional accept if you are weighing it for a workshop paper, and make the revision requirements explicit.

Referee Report

3 major / 7 minor

Summary. The paper proposes a two-stage pipeline for detecting pancreatic ductal adenocarcinoma (PDAC) in 3D abdominal CT. In the first stage, a segmentation network trained only on normal cases produces a binary pancreas mask for each scan. In the second stage, a FusionNet takes both the CT volume and the predicted mask as input and performs binary classification. The fusion is parameterized by the layer α at which two branches are combined and the operation β ∈ {+, *, concatenation}; a grid search over 18 architectures is performed within 4-fold cross-validation on the classification set. The selected architecture, FusionNet3* (α=3, β=multiplication), is reported to achieve a sensitivity of 92.65% and a specificity of 97.00% on 200 normal and 136 PDAC scans, improving over image-only (83.09% sensitivity, 92.00% specificity) and mask-only baselines. The authors also report an oracle combination of image-only and mask-only decisions (Image+Mask GT) that shows large complementarity between the two input modalities.

Significance. If validated, the work is clinically relevant and methodologically interesting. A strength is the oracle control (Image+Mask GT), which directly demonstrates that image-only and mask-only predictions are complementary and that an upper bound of fusion is much higher than either branch. The functional space of fusion architectures is clearly specified, and the authors openly describe the grid search. The method only requires pancreas annotations on normal cases, which is a practical advantage. However, the evaluation is based on a single private dataset, the reported gains are not accompanied by error bars or per-fold results, and the architecture is selected on the same folds used for evaluation. These issues limit the strength of the quantitative claims; nevertheless, the core idea is plausible and worthy of publication after substantial revision.

major comments (3)

[§3.1–3.2, Table 1 and Table 2] The headline results are obtained by selecting the best architecture on the same cross-validation folds used to compute the final metrics. In §3.1 the authors state 'we apply grid search on α and β' over all 18 pairs, and in §3.2 FusionNet3* is presented as the final method with the best F1 in Table 2. No held-out validation set or nested cross-validation is described. Because the same four folds are used both for model selection and for evaluation, the reported sensitivity and specificity are the best of 18 models evaluated on the test folds. With 34 abnormal and 50 normal cases per fold, the reported gains over Image (about 13 abnormal cases and 10 normal cases) are within the range that can arise from selection over many correlated models. Please provide per-fold metrics for all 18 architectures, error bars, and a model-selection procedure that does not use the test folds (e.g., nested cross-validation or a separate validation split), or explicitly frame the reported numbers as an upper bound of the architecture search.
[§2.2 and §3.1] The mask branch is generated by a segmentation network trained only on normal cases (X1) and then applied to both normal and abnormal cases in X2. The paper does not evaluate the quality of the predicted mask on PDAC cases. If masks are systematically corrupted by the presence of the tumor (e.g., deformed or missing pancreas), the shape branch in FusionNet may encode a dataset-specific artifact rather than true shape information, and the complementarity claim could be an artifact of the specific mask errors. Please report mask Dice or visual examples on abnormal cases, or perform an analysis that controls for mask corruption, for instance by ablating with ground-truth masks on a subset of the data.
[§3.2, Table 1] The paper does not provide confidence intervals or repeated-run variability for any of the reported metrics. Given the small dataset (200 normal, 136 abnormal) and a single random 4-fold split, the differences between FusionNet3* and Image (9.56% sensitivity, 5% specificity) may not be statistically significant. Please provide bootstrap confidence intervals or repeated cross-validation with different seeds, and report the variance across folds.

minor comments (7)

[Conclusion] The conclusion states '200 normal patients and 138 patients with PDAC', but the dataset described in §3.1 contains 136 PDAC cases; please correct this inconsistency.
[§3.1] Please report the inclusion criteria for the patients, how the normal cases were confirmed as cancer-free, and whether the study received institutional review board approval; this information is commonly expected for medical imaging studies.
[Figure 2 and Figure 3] The ROC curves and the metric-comparison plots overlap considerably; consider using separate panels or a table with confidence intervals to improve readability.
[§2.3] The loss weight λ is fixed to 0.7 without a sensitivity analysis; please justify this choice or show that the results are stable across a range of λ values.
[§3.2, Table 1] The 'Image+Mask GT' result is an oracle that uses ground-truth labels to decide if either method is correct; please state clearly that this is an idealized upper bound and not achievable in practice, and avoid the phrase 'proves' when describing what it demonstrates.
[§3.1] Please clarify whether the same training hyperparameters (learning rate, decay, number of iterations) were used for all 18 architectures in the grid search, and whether early stopping was applied.
[§2.3, Eq. (1)] The notation w = {w1_{1:α}, w2_{1:α}, w_{α:L}} is ambiguous; please define the shape or dimensionality of each parameter group, and clarify that the two branches have independent weights before fusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: FusionNet3* is an empirically selected architecture, not a quantity derived from its own inputs.

full rationale

The paper makes no derivation claim that could reduce to its own inputs. The central result is an empirical measurement: the FusionNet architecture is selected by grid search over a pre-defined 18-model functional space (Sec. 3.1), and the reported 92.65% sensitivity / 97.00% specificity are the measured test-fold numbers of the selected model, not a quantity computed from a fitted parameter or from a theorem whose conclusion is assumed. The complementarity claim is supported by two independent observations: (i) the oracle 'Image+Mask GT' analysis in Sec. 3.2 is explicitly an upper bound computed using ground-truth labels and is not claimed as an achieved system, and (ii) Table 2 shows that most of the 18 fusion models outperform the single-branch baselines across the same folds. Neither observation is forced by construction. The self-citations are not load-bearing in a circular sense: [11] supplies the segmentation network used as a fixed preprocessing tool trained on X1 and tested on X2, and [5] provides an external baseline (AE+Mask); neither is used to justify the fusion result itself. The grid search being conducted on the same four cross-validation folds that produce the final metric is a real model-selection/statistical-validity concern, but it is a correctness risk rather than circularity: the selected model's performance is still measured, not defined by the selection. No equation, definition, or self-citation chain in the paper makes any claimed prediction equivalent to an input by construction.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central empirical claim rests mainly on the two-stage pipeline design and the small fusion search. The most load-bearing free parameters are the fusion location and operation, both selected on the evaluation folds, which biases the reported performance. No new physical entities are introduced.

free parameters (4)

loss weight λ = 0.7
Weighted cross-entropy loss weight to balance class difference; chosen by hand (Section 2.3).
fusion layer α = 3
Selected by grid search over {1..6} on the same 4-fold CV data; this is model selection fitted to the evaluation folds (Section 3.1).
fusion operation β = *
Selected by grid search over {+,*,⊕} on the same 4-fold CV data (Section 3.1).
learning rate and decay = 0.01, decay 0.9997
Optimizer hyperparameters, chosen by hand (Section 3.1).

assumptions (3)

domain assumption Pancreas shape (from mask) and texture (from CT) carry complementary information for PDAC detection.
This is the paper's motivating premise; the oracle Image+Mask GT analysis supports it empirically on this dataset, but it is assumed for the method design (Section 1).
domain assumption A segmentation network trained only on normal pancreas masks produces masks on abnormal scans that are accurate enough to serve as shape inputs.
The two-stage pipeline relies on g(X) over X2, including 136 PDAC cases the segmentation model never saw during training (Section 2.2).
ad hoc to paper The 18 pre-defined fusion architectures are a representative functional space, and selecting the best on the same CV folds gives a fair performance estimate.
The 'optimal' architecture is chosen by grid search on the same folds used for evaluation; this assumes no selection bias (Section 3.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of FusionNet: Incorporating Shape and Texture for Abnormality Detection in 3D Abdominal CT Scans." pith.science (2026). https://pith.science/paper/ZCZQ352F

@misc{pith2026190807654,
  author       = {Pith},
  title        = {Pith review of: FusionNet: Incorporating Shape and Texture for Abnormality Detection in 3D Abdominal CT Scans},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZCZQ352F}},
  note         = {Machine review of arXiv:1908.07654}
}

read the original abstract

Automatic abnormality detection in abdominal CT scans can help doctors improve the accuracy and efficiency in diagnosis. In this paper we aim at detecting pancreatic ductal adenocarcinoma (PDAC), the most common pancreatic cancer. Taking the fact that the existence of tumor can affect both the shape and the texture of pancreas, we design a system to extract the shape and texture feature at the same time for detecting PDAC. In this paper we propose a two-stage method for this 3D classification task. First, we segment the pancreas into a binary mask. Second, a FusionNet is proposed to take both the binary mask and CT image as input and perform a binary classification. The optimal architecture of the FusionNet is obtained by searching a pre-defined functional space. We show that the classification results using either shape or texture information are complementary, and by fusing them with the optimized architecture, the performance improves by a large margin. Our method achieves a specificity of 97% and a sensitivity of 92% on 200 normal scans and 136 scans with PDAC.

Figures

Figures reproduced from arXiv: 1908.07654 by the authors.

**Figure 1.** The pipeline of our framework. In stage 1, a segmentation network is trained using the normal data. Then the segmentation network is tested on both normal and abnormal data. The 3D mask and image are cropped and scaled as the input of second stage. At the right side, we show the examples of fusion model using different α, β. Note that these three models share the same architecture after layer 3 because α <= 3 in the… view at source ↗

**Figure 2.** ROC curves for comparison of different fusion strategies. Left: fused by +. Mid: fused by ∗. Right: fused by ⊕. The Image, Mask and AE+Mask are the baseline methods without fusing. The Image+Mask GT is the pseudo upper bound of the fusing. SEN SPEC AUC F1 AE+Mask [5] 77.94 91.00 89.04 81.54 Mask 82.35 91.50 92.94 84.53 Image 83.09 92.00 95.95 85.28 Naive Fusion 83.09 95.50 97.17 87.60 FusionNet3*(Ours) 92.65 97.00 9… view at source ↗

**Figure 3.** Comparison on the sensitivity, specificity, AUC and F1 score between different fusion architectures. only the image works the best, which indicates the importance of texture for detecting PDAC. For the other two methods using only shape information, directly training a discriminator achieves better results, showing that the constraint of auto-encoder can harm the classification performance. α 1 2 3 4 5 6 F1 88.97 8… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages

[1]

National Cancer Institute

Seer cancer statistics review 1975-2015. National Cancer Institute. Bethesda, MD

work page 1975
[2]

In: ISICDM (2018)

Chen, X., Chen, Y., Ma, C., Liu, X., Tang, X.: Classiﬁcation of pancreatic tumors based on mri images using 3d convolutional neural networks. In: ISICDM (2018)

work page 2018
[3]

In: ACCV (2017)

Hazirbas, C., Ma, L., Domokos, C., Cremers, D.: Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture. In: ACCV (2017)

work page 2017
[4]

In: NeurIPS (2012)

Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classiﬁcation with deep con- volutional neural networks. In: NeurIPS (2012)

work page 2012
[5]

ArXiv (2018)

Liu, F., Xie, L., Xia, Y., Fishman, E.K., Yuille, A.L.: Joint shape representation and classiﬁcation for detecting pdac. ArXiv (2018)

work page 2018
[6]

Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. pp. 565–571 (10 2016)

work page 2016
[7]

In: MICCAI (2015)

Roth, H.R., Lu, L., Farag, A., Shin, H.C., Liu, J., Turkbey, E.B., Summers, R.M.: Deeporgan: Multi-level deep convolutional networks for automated pancreas seg- mentation. In: MICCAI (2015)

work page 2015
[8]

In: NeurIPS (2017)

Wu, J., Wang, Y., Xue, T., Sun, X., Freeman, W.T., Tenenbaum, J.B.: MarrNet: 3D Shape Reconstruction via 2.5D Sketches. In: NeurIPS (2017)

work page 2017

Show all 17 references

[9]

Xia, Y., Liu, F., Yang, D., Cai, J., Yu, L., Zhu, Z., Xu, D., Yuille, A.L., Roth, H.: 3d semi-supervised learning with uncertainty-aware multi-view co-training. arXiv
[10]

In: MICCAI

Xia, Y., Xie, L., Liu, F., Zhu, Z., Fishman, E.K., Yuille, A.L.: Bridging the gap between 2d and 3d organ segmentation with volumetric fusion net. In: MICCAI
[11]

CVPR (2018)

Yu, Q., Xie, L., Wang, Y., Zhou, Y., Fishman, E.K., Yuille, A.L.: Recurrent saliency transformation network: Incorporating multi-stage visual cues for small organ seg- mentation. CVPR (2018)

2018
[12]

In: MICCAI (2017)

Zhang, L., Lu, L., Summers, R., Kebebew, E., Yao, J.: Personalized pancreatic tumor growth prediction via group learning. In: MICCAI (2017)

2017
[13]

In: WACV (2019)

Zhou, Y., Wang, Y., Tang, P., Bai, S., Shen, W., Fishman, E., Yuille, A.: Semi- supervised 3d abdominal multi-organ segmentation via deep multi-planar co- training. In: WACV (2019)

2019
[14]

In: MICCAI (2017)

Zhou, Y., Xie, L., Fishman, E.K., Yuille, A.L.: Deep supervision for pancreatic cyst segmentation in abdominal ct scans. In: MICCAI (2017)

2017
[15]

In: MICCAI (2017)

Zhou, Y., Xie, L., Shen, W., Wang, Y., Fishman, E.K., Yuille, A.L.: A ﬁxed-point model for pancreas segmentation in abdominal ct scans. In: MICCAI (2017)

2017
[16]

In: 2018 International Conference on 3D Vision (3DV) (2018)

Zhu, Z., Xia, Y., Shen, W., Fishman, E., Yuille, A.: A 3d coarse-to-ﬁne framework for volumetric medical image segmentation. In: 2018 International Conference on 3D Vision (3DV) (2018)

2018
[17]

ArXiv (2018)

Zhu, Z., Xia, Y., Xie, L., Fishman, E.K., Yuille, A.L.: Multi-scale coarse-to-ﬁne segmentation for screening pancreatic ductal adenocarcinoma. ArXiv (2018)

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

National Cancer Institute

Seer cancer statistics review 1975-2015. National Cancer Institute. Bethesda, MD

work page 1975

[2] [2]

In: ISICDM (2018)

Chen, X., Chen, Y., Ma, C., Liu, X., Tang, X.: Classiﬁcation of pancreatic tumors based on mri images using 3d convolutional neural networks. In: ISICDM (2018)

work page 2018

[3] [3]

In: ACCV (2017)

Hazirbas, C., Ma, L., Domokos, C., Cremers, D.: Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture. In: ACCV (2017)

work page 2017

[4] [4]

In: NeurIPS (2012)

Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classiﬁcation with deep con- volutional neural networks. In: NeurIPS (2012)

work page 2012

[5] [5]

ArXiv (2018)

Liu, F., Xie, L., Xia, Y., Fishman, E.K., Yuille, A.L.: Joint shape representation and classiﬁcation for detecting pdac. ArXiv (2018)

work page 2018

[6] [6]

Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. pp. 565–571 (10 2016)

work page 2016

[7] [7]

In: MICCAI (2015)

Roth, H.R., Lu, L., Farag, A., Shin, H.C., Liu, J., Turkbey, E.B., Summers, R.M.: Deeporgan: Multi-level deep convolutional networks for automated pancreas seg- mentation. In: MICCAI (2015)

work page 2015

[8] [8]

In: NeurIPS (2017)

Wu, J., Wang, Y., Xue, T., Sun, X., Freeman, W.T., Tenenbaum, J.B.: MarrNet: 3D Shape Reconstruction via 2.5D Sketches. In: NeurIPS (2017)

work page 2017