Pith. sign in

REVIEW 5 major objections 5 minor 19 references

Deep Learning-Based Breast Cancer Detection in Mammography: A Multi-Center Validation Study in Thai Population

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a deep learning system trained on mammograms from one Thai hospital detects breast cancer with AUROC 0.894 on its home test set, 0.963 on biopsy-confirmed cases, and 0.937 on cases from two other hospitals, and that…

desk verdict A credible, useful first large-scale multi-center validation of mammography AI in a Thai population, but the out-of-domain reference standard is unstated and the acceptance metric is partly self-referential. read the letter →

arxiv 2506.03177 v2 pith:5XPU6N5O submitted 2025-05-29 eess.IV cs.AIcs.CVcs.LG

Isarun Chamveha , Supphanut Chaiyungyuen , Sasinun Worakriangkrai , Nattawadee Prasawang , Warasinee Chaisangmongkon , Pornpim Korpraphong , Voraparee Suvannarerg , Shanigarn Thiravit
show 7 more authors
Chalermdej Kannawat Kewalin Rungsinaporn Suwara Issaragrisil Payia Chadbunchachai Pattiya Gatechumpol Chawiporn Muktabhant Patarachai Sereerat
This is my paper · ORCID
classification eess.IVcs.AIcs.CVcs.LG
keywords BreastcancerMammographyScreeningDeeplearningArtificialintelligenceMulti-centervalidationThaipopulationLesionlocalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a mammography AI trained on images from one large Thai hospital can detect breast cancer in other Thai hospitals, not just at its training site. On the model's held-out home test set it reaches an AUROC of 0.894; on a biopsy-confirmed set, 0.963; and on 761 cases from two other hospitals, 0.937. The system also highlights suspicious lesions, with cancer-level localization fractions between about 0.71 and 0.86 depending on the dataset, and radiologists in the study accepted its outputs 96.7% and 89.3% of the time. The authors present these results as evidence that the system is a reliable AI assistant for mammogram interpretation in Thai clinical practice and that it generalizes to new hospital settings.

What carries the argument

The load-bearing mechanism is the model's joint classification-and-localization architecture. The encoder is EfficientNetV2(S), but its default squeeze-and-excitation attention is replaced with an Attend-and-Compare Module, which compares each region of the image with its surrounding context in a way meant to mimic a radiologist comparing symmetric or related zones. An External Attention layer learns correlations across samples, and a decoder with skip connections enlarges the feature map to help find small scattered calcifications. Probabilistic-CAM pooling then aggregates the features into two simultaneous outputs: a flattened classification score and a spatial heatmap. Because the model is trained with both image-level labels and pixel-level lesion annotations, the same network learns to say whether cancer is present and where it is, which is what lets the paper evaluate both detection and localization.

What would settle it

Take a new cohort of screening mammograms from a hospital the model has never seen, determine cancer status only by biopsy or two-year follow-up, and compare the model's AUROC on that pathology-grounded cohort against the reported values; if it falls substantially below 0.937, the claim of generalizable, clinically reliable detection fails.

Watch

Extended reading notes

Core claim

The authors claim that a single deep learning model can both classify and localize breast lesions in mammography for a Thai population. The architecture is a modified EfficientNetV2(S) encoder with an Attend-and-Compare attention module in place of the default attention, an external attention layer, a decoder with skip connections, and Probabilistic-CAM pooling that produces both a classification score and a heatmap. Trained on 58,022 mammograms from one large Thai medical center and validated on three sets—9,421 held-out in-domain cases (AUROC 0.894), 883 biopsy-confirmed cases (AUROC 0.963), and 761 out-of-domain cases from two other hospitals (AUROC 0.937)—the model detects calcification, mass, axillary adenopathy, and architectural distortion, and localizes them with lesion localization fractions from 0.461 (other lesions, out-of-domain) to 0.861 (cancer overall, biopsy-confirmed). Concordance with radiologist reports is 83.5% and 78.1% for classification and 84.0% and 79.6% for localization on the two clinical datasets, and radiologists accept the AI's results 96.7% and 89.3% of the time. The conclusion the paper draws is that the system is a dependable AI assistant for mammogram interpretation and generalizes across Thai hospital settings.

Load-bearing premise

The results depend on the assumption that the radiology-report labels used to define cancer in the main in-domain test set—written at the same hospital where the model was trained—are an accurate measure of true cancer status; if those labels reflect report style rather than actual disease, the reported AUROC would overstate real-world performance.

Editorial extensions

If this is right

  • If the reported out-of-domain AUROC of 0.937 holds in routine use, the system could act as a second reader in Thai screening programs, flagging cases that need a radiologist's closer look.
  • With sensitivity around 93% at the chosen operating point on out-of-domain data and acceptance rates of 96.7% and 89.3%, most AI findings would not require a change to the radiologist's read, meaning the main benefit is catching cases that might otherwise be missed.
  • Because the model produces heatmaps with lesion localization fractions of roughly 0.71 to 0.86 for cancer overall, it can direct the reader to the suspicious region instead of requiring a whole-image search.
  • Concordance rates of 78-84% between the AI and radiology reports imply that in about one in five cases the AI and human disagree, making the system better suited as a prompting tool for double-reading than as a standalone replacement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The in-domain AUROC of 0.894 is the figure most likely to be optimistic, because its reference standard is radiology reports from the same hospital where the model was trained; the strongest evidence of generalizability rests on just 761 out-of-domain cases, so a larger prospective cohort is the natural next test.
  • Inference: The paper does not stratify performance by breast density, and extremely dense breasts make up only about 5-6% of the validation sets; since dense tissue is common and makes cancer harder to see, a density-stratified analysis would be a meaningful extension.
  • Inference: The concordance definition counts 'edit' and 'add' as agreement, so the reported 78-84% agreement is not all-or-nothing matching; a stricter agreement metric would produce lower numbers and should be reported for clinical deployment decisions.
  • Inference: A practical extension the authors do not quantify is workflow economics: with specificity between 58% and 76% depending on the dataset, using the AI as a triage tool would generate a certain number of extra recalls or workups, and the net effect on radiologist workload depends on that false-positive rate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript describes a deep learning system (Inspectra MMG) for breast cancer detection in mammography, trained on 58,022 mammograms from Siriraj Hospital and evaluated on three validation sets: an in-domain held-out set (9,421 cases), a biopsy-confirmed set (883 cases), and an out-of-domain set (761 cases from two other hospitals). The authors report AUROC values of 0.894, 0.963, and 0.937 on these sets, along with lesion-localization metrics (LLF/NLF), concordance rates with radiologist reports (83.5% and 78.1% classification), radiologist acceptance rates (96.7% and 89.3%), and System Usability Scale scores (74.17 and 69.20). The paper claims that the system is a reliable AI assistant for mammogram interpretation in Thai clinical settings.

Significance. If the results are accurate, this is a valuable contribution to the literature on AI in mammography, providing the first large-scale validation for a Thai population and including a multi-center out-of-domain set and a biopsy-confirmed reference set. The biopsy-confirmed set with pathology and two-year follow-up is a particular strength, and the evaluation of localization and clinical usability goes beyond simple classification metrics. However, several methodological issues—most importantly the unverified reference standard for the out-of-domain set and the partly tautological acceptance-rate calculation—need to be resolved before the generalizability and clinical-acceptance claims can be accepted.

major comments (5)
  1. [Section 4.3, Table 3] The reference standard for the 761-case generalization set is not described. The paper does not state whether the labels come from pathology, follow-up, adjudicated expert review, or original radiology reports; Table 3 reports BIRADS and abnormality categories but not label provenance. Since the out-of-domain AUROC (0.937) is the primary evidence for multi-center generalizability, this omission is critical. If the labels are derived from unverified radiology reports, the result may reflect agreement with reporting style rather than true cancer status. Please specify the labeling process, any independent verification, and support it with outcome data if available.
  2. [Section 3.3, Tables 5/7/9] The use of the Clopper-Pearson method for AUC confidence intervals is statistically invalid. Clopper-Pearson is an exact interval for binomial proportions; the AUC is a rank-based statistic and its sampling distribution is not binomial. The reported AUC intervals in Tables 5, 7, and 9 should be recomputed with a rank-based method (e.g., DeLong's test or bootstrap), and the methods section corrected accordingly. The Clopper-Pearson intervals for sensitivity and specificity may be retained, but the sentence 'Clopper-Pearson method was used for estimating 95% CIs for AUC' is incorrect as written.
  3. [Section 5.2, Tables 15/16] The radiologist acceptance rate is inflated by automatically counting all 'Agree' concordance cases as accepted (423 of 883 and 307 of 761). Because 'Agree' is defined in Table 11 as 'Radiologist report completely matches with AI,' these cases are accepted by definition whenever the AI agrees with the report, making the resulting 96.7% and 89.3% acceptance rates partly tautological. Please report the acceptance rate based on radiologist review of all cases, or at minimum provide the acceptance rate for the manually reviewed subset separately from the automatically accepted subset.
  4. [Section 5.1, Tables 11/12/13/14] The concordance definition counts 'Add' cases as concordant. An 'Add' case is one where the radiology report contains findings not detected by the AI (Table 11) or where the AI heatmap misses lesions (Table 12); treating these as concordant means that missed lesions are counted as agreement. This inflates the concordance rates, particularly for the generalization set, where 'Add' comprises 7.2% (classification) and 6.8% (localization) of cases. Report 'Agree,' 'Edit,' 'Add,' and 'Reject' separately, and do not pool 'Add' with true agreement unless a clear clinical justification is provided.
  5. [Section 4.1] The sentence 'All values are computed based on the optimal operating point of the model' is ambiguous but suggests that the operating point (threshold) may have been selected using the test set. If so, the reported sensitivity and specificity are optimistically biased. Please state how the threshold was chosen (e.g., fixed a priori, selected on the validation split only) and, if it was optimized on the test set, re-estimate sensitivity and specificity using a pre-specified threshold.
minor comments (5)
  1. [Section 3.1, Table 1] The sum of the 'Condition' categories is 70,504, which exceeds the reported dataset size of 58,022; clarify whether cases can have multiple conditions and provide mutually exclusive counts or explain the overlap.
  2. [Section 4.1] The in-domain test set is described as containing 9,421 mammograms, but a 15% split of 58,022 would be approximately 8,703; clarify whether the 58,022 figure refers to images and the 9,421 to examinations or vice versa.
  3. [Tables 6/8/10] The total lesion counts for the 'Cancer' row do not match the sum of the per-type rows (e.g., in Table 6, 1,879 vs. 1,917); reconcile these numbers or state that 'Cancer' includes only a subset of lesions.
  4. [Section 5.2] Treating a 'neutral' score of 2 as acceptable may overstate acceptance; consider reporting the full distribution of scores (1-4) and the percentage rated 3 or 4.
  5. [Section 3.2] The reference standard for the development set is described as based on radiology reports; please state whether any pathology confirmation was available for a subset of the development data, and whether the same labeling protocol was applied to the in-domain test set.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity in detection results; one minor definitional shortcut in the radiologist-acceptance metric.

  1. self definitional [Section 5.2, 'Usefulness of the MMG Model', paragraph beginning 'To mitigate their workload' (Tables 15 and 16).]
    "To mitigate their workload, we excluded the 423 cases which were categorized as 'Agree' in both classification and localization concordance tests. These cases are considered as 'Accept' in the final acceptance score calculation since they represent cases that are correct in both classification and localization."

    The reported acceptance rate is defined so that cases already labeled concordant are automatically counted as accepted without any radiologist rating. Thus the 96.7% and 89.3% acceptance figures are not purely independent measurements of radiologist acceptance: 423 of 883 (47.9%) biopsy-confirmed cases and 307 of 761 (40.3%) generalization cases enter the numerator by definition rather than by a 1-4 usefulness score. The acceptance result is therefore partly equivalent to the concordance result, by construction. This affects only the secondary usability/acceptance claim; the central AUROC and localization results are evaluated against pathology or report-derived labels and are not reduced by this step.

full rationale

The paper's central detection claims (AUROC 0.894 on the in-domain set, 0.963 on the biopsy-confirmed set, 0.937 on the generalization set) come from held-out evaluation against labels that are not fitted parameters of the model. The biopsy-confirmed set uses pathology plus two-year follow-up for benign/normal cases, providing external grounding. The in-domain test set shares the report-labeling convention used in training, which is a generalizability caveat but not a circular reduction because the test cases are not used to fit the model. The out-of-domain reference-standard provenance is underdescribed, which is a correctness risk rather than a circularity. The one construction-dependent result is the radiologist acceptance metric in Section 5.2: concordant 'Agree' cases are auto-accepted, so the acceptance rate is partly definitional. This is explicitly disclosed and does not affect the main detection derivation, warranting a low score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests primarily on the accuracy of the radiological labels used for training and in-domain testing, and on the statistical methods used to compute confidence intervals. The model itself is an engineering artifact; no new physical or theoretical entities are introduced.

free parameters (2)
  • Benign probability cap = 15%
    Section 3.4 limits the maximum probability for benign lesions to 15% to avoid overemphasizing them; this is a hand-set threshold affecting output scores and may influence operating point.
  • Operating point threshold = not specified
    Sensitivity and specificity are reported at the model's optimal operating point (Section 4.1); this threshold is chosen post hoc to balance sensitivity and specificity, affecting reported metrics.
assumptions (4)
  • domain assumption Radiology reports used for training and in-domain test labels are accurate representations of true disease status
    Section 3.2 states labels were extracted from diagnostic reports and verified by radiologists; if reports are biased, the learned labels are biased.
  • domain assumption The three lesion categories (calcification, mass, other) are sufficient to represent clinically significant mammographic abnormalities
    Table 2 maps all output nodes to these three final categories; if a cancer presents as a different pattern, the model may miss it.
  • standard math Clopper-Pearson confidence intervals are valid for AUROC
    Section 3.3 claims Clopper-Pearson was used for AUC CIs; this is a statistical assumption and is not standard for AUC.
  • domain assumption The out-of-domain dataset is representative of the broader Thai screening population
    Generalizability is inferred from 761 cases from two hospitals; this assumes these cases reflect the range of imaging protocols and patient characteristics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep Learning-Based Breast Cancer Detection in Mammography: A Multi-Center Validation Study in Thai Population." pith.science (2026). https://pith.science/paper/5XPU6N5O

@misc{pith2026250603177,
  author       = {Pith},
  title        = {Pith review of: Deep Learning-Based Breast Cancer Detection in Mammography: A Multi-Center Validation Study in Thai Population},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5XPU6N5O}},
  note         = {Machine review of arXiv:2506.03177}
}
read the original abstract

This study presents a deep learning system for breast cancer detection in mammography, developed using a modified EfficientNetV2 architecture with enhanced attention mechanisms. The model was trained on mammograms from a major Thai medical center and validated on three distinct datasets: an in-domain test set (9,421 cases), a biopsy-confirmed set (883 cases), and an out-of-domain generalizability set (761 cases) collected from two different hospitals. For cancer detection, the model achieved AUROCs of 0.89, 0.96, and 0.94 on the respective datasets. The system's lesion localization capability, evaluated using metrics including Lesion Localization Fraction (LLF) and Non-Lesion Localization Fraction (NLF), demonstrated robust performance in identifying suspicious regions. Clinical validation through concordance tests showed strong agreement with radiologists: 83.5% classification and 84.0% localization concordance for biopsy-confirmed cases, and 78.1% classification and 79.6% localization concordance for out-of-domain cases. Expert radiologists' acceptance rate also averaged 96.7% for biopsy-confirmed cases, and 89.3% for out-of-domain cases. The system achieved a System Usability Scale score of 74.17 for source hospital, and 69.20 for validation hospitals, indicating good clinical acceptance. These results demonstrate the model's effectiveness in assisting mammogram interpretation, with the potential to enhance breast cancer screening workflows in clinical practice.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

  1. [1]

    International Agency for Research on Cancer

    Breast Cancer. International Agency for Research on Cancer. Accessed on July 15, 2024. https://www.iarc.who.int/cancer-type/breast-cancer/

    work page 2024

  2. [2]

    Informatics in Medicine Unlocked 41, 101317 (2023)

    Martinez, R.G., Dongen, D.M.: Deep learning algorithms for the early detection of breast cancer: A comparative study with traditional machine learning. Informatics in Medicine Unlocked 41, 101317 (2023)

    work page 2023

  3. [3]

    Scientific Reports 8(1), 1–8 (2018)

    Kim, E.K., Kim, H.-E., Han, K., Kang, B.J., Sohn, Y.-M., Woo, O.H., Lee, C.W.: Applying data-driven imaging biomarker in mammography for breast cancer screening: preliminary study. Scientific Reports 8(1), 1–8 (2018)

    work page 2018

  4. [4]

    Journal of Digital Imaging 30(4), 499–505 (2017)

    Teare, P., Fishman, M., Benzaquen, O., Toledano, E., Elnekave, E.: Malignancy detection on mammography using dual deep convolutional neural networks and genetically discovered false color input enhancement. Journal of Digital Imaging 30(4), 499–505 (2017)

    work page 2017

  5. [5]

    Food and Drug Administration: 510(k) premarket notification: Health- Mammo

    U.S. Food and Drug Administration: 510(k) premarket notification: Health- Mammo. 510(k) Clearance K200905, U.S. Food and Drug Administration, Silver Spring, MD (2020). Cleared July 16, 2020. https://www.accessdata.fda.gov/ scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K200905

    work page 2020

  6. [6]

    Lunit INSIGHT MMG, an AI Solution for Breast Can- cer Detection, Now CE Certified. Lunit. Accessed on Jan- uary 2, 2021 (2020). https://www.lunit.io/en/company/news/ lunit-insight-mmg-an-ai-solution-for-breast-cancer-detection-now-ce-certified

    work page 2020

  7. [7]

    In: 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), pp

    Guan, S., Loew, M.: Analysis of generalizability of deep neural networks based on the complexity of decision boundary. In: 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 101–106 (2020). IEEE

    work page 2020

  8. [8]

    Korean Journal of Radiology 22(8), 1225–1239 (2021)

    Yoon, J.H., Kim, E.-K.: Deep learning-based artificial intelligence for mammog- raphy. Korean Journal of Radiology 22(8), 1225–1239 (2021)

    work page 2021

Show all 19 references
  1. [9]

    The Lancet Digital Health 2(3), 138–148 (2020)

    Kim, H.-E., Kim, H.H., Han, B.-K., Kim, K.H., Han, K., Nam, H., Lee, E.H., Kim, E.-K.: Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study. The Lancet Digital Health 2(3), 138–148 (2020)

    2020

  2. [10]

    JAMA Network Open 3(3), 200265 (2020)

    Schaffter, T., Buist, D.S., Lee, C.I., et al.: Evaluation of combined artificial intel- ligence and radiologist assessment to interpret screening mammograms. JAMA Network Open 3(3), 200265 (2020)

    2020

  3. [11]

    In: Interna- tional Conference on Machine Learning, pp

    Tan, M., Le, Q.: Efficientnetv2: Smaller models and faster training. In: Interna- tional Conference on Machine Learning, pp. 10096–10106 (2021). PMLR 21

    2021

  4. [12]

    Breast Cancer Research 17(1), 147 (2015)

    Brentnall, A.R., Harkness, E.F., Astley, S.M., et al.: Mammographic density adds accuracy to both the tyrer-cuzick and gail breast cancer risk models in a prospective uk screening cohort. Breast Cancer Research 17(1), 147 (2015)

    2015

  5. [13]

    Medical Physics 35(2), 435–445 (2008)

    Chakraborty, D.P., Yoon, H.-J.: Operating characteristics predicted by models for diagnostic tasks involving lesion localization. Medical Physics 35(2), 435–445 (2008)

    2008

  6. [14]

    arXiv preprint arXiv:2005.14480 (2020)

    Ye, W., Yao, J., Xue, H., Li, Y.: Weakly Supervised Lesion Localization With Probabilistic-CAM Pooling. arXiv preprint arXiv:2005.14480 (2020)

    2020 arXiv

  7. [15]

    In: European Conference on Computer Vision, pp

    Kim, M., Park, J., Na, S., Park, C.M., Yoo, D.: Learning visual context by comparison. In: European Conference on Computer Vision, pp. 576–592 (2020). Springer

    2020

  8. [16]

    OpenCV Documentation

    OpenCV Connected Components. OpenCV Documentation. Accessed on July 15, 2024. https://docs.opencv.org/3.4/d3/dc0/group imgproc shape.html# gaedef8c7340499ca391d459122e51bef5

    2024

  9. [17]

    PLoS One 17(2), 0264383 (2022)

    Kim, E.Y., et al.: Concordance rate of radiologists and a commercialized deep- learning solution for chest x-ray: Real-world experience with a multicenter health screening cohort. PLoS One 17(2), 0264383 (2022)

    2022

  10. [18]

    NPJ Digital Medicine 4(1), 31 (2021)

    Gaube, S., et al.: Do as ai say: susceptibility in deployment of clinical decision- aids. NPJ Digital Medicine 4(1), 31 (2021)

    2021

  11. [19]

    Journal of Usability Studies 8(2), 29–40 (2013) 22

    Brooke, J.: Sus: a retrospective. Journal of Usability Studies 8(2), 29–40 (2013) 22

    2013

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.