Improving localization-based approaches for breast cancer screening exam classification

Jason Phang; Krzysztof J. Geras; Kyunghyun Cho; Linda Moy; Nan Wu; S. Gene Kim; Thibault F\'evry

REVIEW 3 major objections 4 minor 12 references

Improving localization-based approaches for breast cancer screening exam classification

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper reports that a localization-based deep CNN, trained on over 200,000 screening exams, predicts breast malignancy with an AUC of 0.919 and generates bounding boxes for findings, cutting the baseline error by 23 percent.

desk verdict Solid empirical extension of localization-based mammography CAD with an honest ablation table; the headline AUC should be read as 'negative vs. biopsy' performance rather than pure cancer detection. read the letter →

arxiv 1908.00615 v1 pith:L2LGE56U submitted 2019-08-01 eess.IV cs.CVstat.ML

Thibault Févry , Jason Phang , Nan Wu , S. Gene Kim , Linda Moy , Kyunghyun Cho , Krzysztof J. Geras This is my paper · ORCID

classification eess.IVcs.CVstat.ML

keywords breastcancerscreeningmammographydeeplearningobjectdetectionlocalizationAUCcomputer-aideddiagnosisinterpretability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports that an object-detection network trained directly on radiologist-annotated lesion locations can classify whole screening exams for breast malignancy and, at the same time, point at the suspicious findings. On a test set drawn from over 200,000 exams, the model reaches an AUC of 0.919, a 23 percent relative reduction in error compared with the earlier classification baseline, and its boxes mark both benign and malignant lesions. The authors' central claim is that localization-based training does not sacrifice classification performance and yields interpretable predictions, which matters because a radiologist can see why the model flags a breast.

What carries the argument

The operative mechanism is a two-stage detection network composed of a region proposal network and a classifier head, built on a feature pyramid backbone and trained at high image resolution. Annotations from biopsied exams are converted to bounding boxes; exams without annotations enter training as negatives to limit overfitting. Inference takes, for each breast, the maximum malignant score among boxes in each of the two views and averages those maxima. The paper also relaxes the intersection-over-union threshold for foreground proposals and lowers the detection score threshold at inference, reflecting the assumption that mammographic findings are sparse and annotations are imprecise.

What would settle it

A held-out set in which every breast has a verified outcome — either a biopsy or a cancer-free follow-up years later — would settle the claim: the model's AUC should stay near 0.919, and its malignant boxes should land on the verified cancers. If AUC collapses toward chance or boxes miss verified malignancies, the result is an artifact of biopsy-only labeling.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that a two-stage object detector trained on pixel-level annotations of biopsied lesions outperforms the previous classification-based system on the same dataset. The best single configuration reaches 0.891 AUC; averaging over multiple random seeds and backbone variants gives 0.919 on the test set, and combining this ensemble with the earlier model's ensemble reaches 0.930. On the reader-study subset, the localization ensemble roughly matches the earlier model's ensemble (0.879 versus 0.876), and ensembling the two reaches 0.895, above the average radiologist's 0.778. The detector also emits bounding boxes, so its predictions are not just scores but visible regions.

Load-bearing premise

The model's labels are only as trustworthy as the rule that any exam without a biopsy within 120 days is negative and any biopsied lesion's pixel annotation is complete and accurate; if hidden cancers exist in non-biopsied breasts, training suppresses true findings and the reported AUC is computed against noisy ground truth.

Editorial extensions

If this is right

Because the localizing ensemble reaches 0.919 AUC on the test set, a detector can compete with and beat a classification-only model, so detection and exam-level diagnosis do not need to be separate systems.
Ensembling the localization model with the earlier classification ensemble raises test AUC to 0.930 and reader-study AUC to 0.895, so the two approaches carry partially complementary information.
The generated bounding boxes give radiologists a visual basis for trusting or doubting a prediction, moving computer-aided screening from opaque scores toward interpretable outputs.
The best single network uses a moderate backbone at high resolution; larger backbones had to run at lower resolution and performed competitively, suggesting resolution and capacity are interchangeable to a degree.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the paper leaves implicit: because non-biopsied exams are labeled negative, a breast with an undetected cancer is used as a negative example; if such cancers are common, the model is being trained to suppress some true malignant findings, and the reported AUC may understate its ability to rank verified cancers.
A testable extension would track the benign boxes the model produces: comparing those boxes against cancers diagnosed in later rounds of screening would show whether the detector is learning general precursors or merely mimicking biopsy-driven labels.
The biopsy-ratio experiments suggest training-set composition is a lever; a curriculum that upsamples high-risk or later-confirmed exams could push the AUC further without architectural change.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a competent extended abstract from the NYU group showing that a Faster R-CNN with FPN and a ResNeXt-101 backbone, trained on the NYU screening dataset, beats their own earlier Wu et al. baseline by 23% relative error reduction on the test set. The result is plausible and the paper is transparent about what makes it work.

What's new: not the architecture—Ribli and Morrell already used detection networks for mammography—but the systematic comparison on a large proprietary dataset, the use of all non-annotated images as negatives, the inference-time pooling over views, and the ensemble. The ablation table is actually informative: biopsy sampling ratio, classifier loss weight, bounding box scaling, resolution/backbone tradeoff. That's the kind of practical detail that saves other groups months of trial and error.

Where it's soft: the headline ensemble AUC (0.919) has no error bars or significance test against the 0.895 baseline; given the variance across runs shown in Table 1, that gap could be more fragile than it looks. More substantively, the labels are defined by biopsy within 120 days, so non-biopsied exams are treated as negative in both training and evaluation. The stress-test note is right that this can reward suppressing true findings that weren't biopsied. But it's not a fatal flaw: the baseline shares the same label definition, so the relative comparison is fair, and the authors openly state in Section 4 that their model is better at separating negative cases from biopsy-needing cases than at separating benign from malignant within biopsied exams. That's an honest and clinically relevant distinction. The bigger issue is the single proprietary test set with no external validation—any AUC needs independent confirmation.

Citation pattern is fine: they cite Ribli, Morrell, and their own prior work appropriately. No invented entities, no missing references that I can spot.

Who should read this: people working on mammography CAD, especially those interested in detection-based approaches and training tricks. It's an extended abstract; it doesn't need a full journal re-referee, but the empirical claims deserve scrutiny with proper error bars and external data.

My recommendation: worth a serious referee, conditional on the authors adding error bars for the ensemble and being explicit about the label-noise limitation. If you're on the program committee, I'd send it to review rather than desk reject.

Referee Report

3 major / 4 minor

Summary. This extended abstract (MIDL 2019) describes a localization-based deep CNN for breast cancer screening exam classification. The model is a Faster R-CNN with a Feature Pyramid Network and ResNet/ResNeXt backbones, trained on over 200,000 screening exams from the NYU dataset. The authors report a test-set AUC of 0.919 for the ensemble, a 23% relative error reduction over the prior baseline of Wu et al. (2019a), and they show that the model generates bounding boxes for benign and malignant findings. The paper also presents ablations over biopsy sampling ratio, classifier loss weighting, bounding box scaling, learning rate schedule, and backbone choice, with evaluation on a reader-study subpopulation.

Significance. If the reported result holds, the paper makes a useful contribution to mammography CAD by showing that an object-detection formulation can improve screening-level malignancy prediction while providing interpretable localization. The work is grounded in a large real-world dataset and includes a reasonable ablation study. The interpretability aspect (bounding boxes) is clinically attractive. However, the headline improvement is not accompanied by uncertainty quantification, and the negative-label definition based on a 120-day biopsy window is a potential source of label noise that could affect both training and evaluation. These issues limit the strength of the central claim as presented.

major comments (3)

[Table 1 and Section 4] The headline ensemble AUC of 0.919 is reported without confidence intervals or a significance test against the Wu et al. ensemble AUC of 0.895. Given the authors themselves note high checkpoint-to-checkpoint variance (Section 5) and report standard deviations of 0.003–0.014 for individual runs, the 23% relative error reduction may not be statistically reliable. Please report bootstrap confidence intervals for the ensemble AUC or a paired significance test, and state how many models compose each ensemble.
[Sections 2–3] The treatment of all exams without a biopsy within 120 days as negative, for both training and test labels, means that screening-detected cancers with delayed or missed biopsy are labeled negative. The model is consequently trained to suppress true findings, and the reported AUC rewards such suppression. The paper's own observation in Section 4 that the model separates negative cases from biopsy-needed cases better than benign from malignant within biopsied cases is consistent with this bias. Please add an explicit discussion of this limitation and, if feasible, validate a subset of the test set with longer follow-up (e.g., one-year follow-up or cancer registry linkage) to estimate the impact on the reported AUC.
[Section 3 (Inference)] The inference score threshold is lowered from 0.05 to 0.001 because the authors found it helps on the validation set. This is one of several hyperparameters (RPN IoU threshold, NMS IoU threshold, resolution per backbone, biopsy sampling ratio) that appear to have been selected based on validation-set performance. The paper does not clarify how many such validation-based decisions were made, nor whether the test set was used only once. If multiple validation-based choices were made using the same validation set, the reported test AUC may be optimistically biased. Please clarify the model-selection protocol.

minor comments (4)

[Abstract] The sentence 'the models generates bounding boxes' contains a subject-verb agreement error; it should be 'the model generates'.
[Section 3 (Optimization)] The text states that learning rate schedules were generally not tuned according to Goyal et al., yet Table 1 includes an ablation row '+ recommended lr schedule'. Please clarify whether this ablation was run subsequently or whether the statement refers to the base setup only.
[Section 5] The claim that validation metrics 'varied significantly between checkpoints' would be more informative if quantified (e.g., range or standard deviation across checkpoints for a representative experiment), since it motivates the ensembling strategy.
[Figure 1] The caption notes that the model predicts a benign lesion with low probability (0.36) without a ground-truth annotation; it would be helpful to state whether such unannotated findings were common and how the authors interpret them given the label-noise concern.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported AUC is an empirical measurement against a published baseline on a shared test set, with no derivation step equivalent to its inputs.

full rationale

The paper's central claim is an experimental comparison: its localization-based model reaches test-set AUC 0.919 versus 0.895 for Wu et al. (2019a), a 23% relative error reduction. This is a measurement, not a derivation. No equation in the paper defines the output in terms of the input or fits a parameter and then reports that fit as a prediction. The baseline, while from the same research group, is a prior public model evaluated on the same test set; comparing against it does not reduce the result to the paper's own assumptions. Self-citations to the NYU dataset (Wu et al., 2019b) and prior work on high resolution (Geras et al., 2017) are contextual and not load-bearing in a circular sense. The use of non-annotated images as negatives and the biopsy-within-120-days label definition are legitimate data-validity concerns about label noise, which the paper itself partially acknowledges in Section 4 when it notes the model distinguishes negative cases from biopsy-needing cases better than benign from malignant within biopsied cases. That limitation affects interpretation of the AUC, but it is not circularity: the reported performance is still an empirical result, not a quantity forced by construction. No fitted parameter is renamed as a prediction, and no uniqueness theorem or prior result is invoked to forbid alternatives. Therefore no circular step is present.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on the quality of the biopsy-based ground truth and the lesion annotations, plus a set of hand-tuned detection hyperparameters. No new physical or architectural entities are introduced; the model uses existing components (Faster R-CNN, FPN, ResNet/ResNeXt backbones).

free parameters (5)

Inference score threshold = 0.001
Lowered from 0.05 to 0.001 because the tail of the score distribution is important when computing AUC (Section 3, Inference). Chosen empirically on validation.
RPN foreground IoU threshold = 0.5
Relaxed from 0.7 because annotations are noisy and scarce (Section 3, IoU thresholds), following Ribli et al. (2018).
NMS IoU threshold = 0.1
Set to 0.1 following Ribli et al. (2018), based on the claim that overlapping detections are less common in mammograms (Section 3, IoU thresholds).
Input image resolution per backbone = R-50: 2200x3000; R-101: 1700x2700; X-101: 1300x2100
Chosen to fit memory constraints while preserving detail; R-101 and X-101 were not evaluated at the R-50 resolution (Section 3, Resolution).
Biopsy sampling ratio and classifier loss weight = variants: ratio 0.75/1, classifier weight 1x/5x
These were varied across runs and the final ensemble averages all runs; no single value is central to the ensemble result (Table 1).

assumptions (3)

domain assumption Biopsy within 120 days of screening is a reliable indicator of malignancy; exams without biopsy are negative.
Used to define labels for training and evaluation in Section 2 (Data) and Section 3 (Use of non-annotated images). False negatives would corrupt both training and AUC measurement.
domain assumption Radiologist pixel-level annotations of biopsied lesions are complete and accurate enough to train a detector.
Bounding boxes are derived from these annotations (Section 3). Noisy or incomplete boxes would degrade the region proposal and classifier training.
domain assumption ImageNet-pretrained backbones transfer useful features to mammography.
All backbones are initialized with ImageNet weights (Section 3). This is a standard practice but an unverified assumption for this domain.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving localization-based approaches for breast cancer screening exam classification." pith.science (2026). https://pith.science/paper/L2LGE56U

@misc{pith2026190800615,
  author       = {Pith},
  title        = {Pith review of: Improving localization-based approaches for breast cancer screening exam classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L2LGE56U}},
  note         = {Machine review of arXiv:1908.00615}
}

read the original abstract

We trained and evaluated a localization-based deep CNN for breast cancer screening exam classification on over 200,000 exams (over 1,000,000 images). Our model achieves an AUC of 0.919 in predicting malignancy in patients undergoing breast cancer screening, reducing the error rate of the baseline (Wu et al., 2019a) by 23%. In addition, the models generates bounding boxes for benign and malignant findings, providing interpretable predictions.

Figures

Figures reproduced from arXiv: 1908.00615 by the authors.

**Figure 1.** An example of an annotation of a malignant lesion (left) and predictions (right) from a X-101 model on a test set breast. Our validation metrics varied significantly between checkpoints. We believe that this is due to (i) the interaction of the components of the training loss, (ii) components of the training loss being only loosely related to the final metric, (iii) a small batch size rendering optimization unstabl… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

[1]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database . In CVPR, 2009

work page 2009
[2]

High-resolution breast cancer screening with multi-view deep convolutional neural networks

Krzysztof J Geras, Stacey Wolfson, Yiqiu Shen, S Kim, Linda Moy, and Kyunghyun Cho. High-resolution breast cancer screening with multi-view deep convolutional neural networks. arXiv:1703.07047, 2017

arXiv 2017
[3]

Accurate, large minibatch sgd: Training imagenet in 1 hour

Priya Goyal, Piotr Doll \'a r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv:1706.02677, 2017

arXiv 2017
[4]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016

2016
[5]

Feature pyramid networks for object detection

Tsung-Yi Lin, Piotr Doll \'a r, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017

2017
[6]

maskrcnn-benchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch

Francisco Massa and Ross Girshick. maskrcnn-benchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch . https://github.com/facebookresearch/maskrcnn-benchmark, 2018

work page 2018
[7]

Large-scale mammography cad with deformable conv-nets

Stephen Morrell, Zbigniew Wojna, Can Son Khoo, Sebastien Ourselin, and Juan Eugenio Iglesias. Large-scale mammography cad with deformable conv-nets. In Image Analysis for Moving Organ, Breast, and Thoracic Images. 2018

work page 2018
[8]

Faster r-cnn: Towards real-time object detection with region proposal networks

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015

2015

Show all 12 references

[9]

Detecting and classifying lesions in mammograms with deep learning

Dezs o Ribli, Anna Horv \'a th, Zsuzsa Unger, P \'e ter Pollner, and Istv \'a n Csabai. Detecting and classifying lesions in mammograms with deep learning. Scientific Reports, 2018

2018
[10]

N. Wu, J. Phang, J. Park, Y. Shen, Z. Huang, M. Zorin, S. Jastrz e bski , T. F \'e vry , J. Katsnelson, E. Kim, S. Wolfson, U. Parikh, S. Gaddam, L. L. Y. Lin, K. Ho, J. D. Weinstein, B. Reig, Y. Gao, H. Toth, K. Pysarenko, A. Lewin, J. Lee, K. Airola, E. Mema, S. Chung, E. Hw...

1903 arXiv
[11]

Gene Kim, Laura Heacock, Linda Moy, Kyunghyun Cho, and Krzysztof J

Nan Wu, Jason Phang, Jungkyu Park, Yiqiu Shen, S. Gene Kim, Laura Heacock, Linda Moy, Kyunghyun Cho, and Krzysztof J. Geras. The NYU breast cancer screening dataset v1.0. Technical report, 2019 b . Available at https://cs.nyu.edu/ kgeras/reports/datav1.0.pdf

2019
[12]

Aggregated residual transformations for deep neural networks

Saining Xie, Ross Girshick, Piotr Doll \'a r, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database . In CVPR, 2009

work page 2009

[2] [2]

High-resolution breast cancer screening with multi-view deep convolutional neural networks

Krzysztof J Geras, Stacey Wolfson, Yiqiu Shen, S Kim, Linda Moy, and Kyunghyun Cho. High-resolution breast cancer screening with multi-view deep convolutional neural networks. arXiv:1703.07047, 2017

arXiv 2017

[3] [3]

Accurate, large minibatch sgd: Training imagenet in 1 hour

Priya Goyal, Piotr Doll \'a r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv:1706.02677, 2017

arXiv 2017

[4] [4]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016

2016

[5] [5]

Feature pyramid networks for object detection

Tsung-Yi Lin, Piotr Doll \'a r, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017

2017

[6] [6]

maskrcnn-benchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch

Francisco Massa and Ross Girshick. maskrcnn-benchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch . https://github.com/facebookresearch/maskrcnn-benchmark, 2018

work page 2018

[7] [7]

Large-scale mammography cad with deformable conv-nets

Stephen Morrell, Zbigniew Wojna, Can Son Khoo, Sebastien Ourselin, and Juan Eugenio Iglesias. Large-scale mammography cad with deformable conv-nets. In Image Analysis for Moving Organ, Breast, and Thoracic Images. 2018

work page 2018

[8] [8]

Faster r-cnn: Towards real-time object detection with region proposal networks

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015

2015