{"id":"d61a4eea-9307-4ca6-950a-d1c4f4461a96","arxiv_id":"2509.05004","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper reports 99.7% accuracy from a ResNet-18 model on breast ultrasound images, but methodological gaps make the result unverifiable.","lead":"This paper applies standard image classification models to breast ultrasound datasets and reports near-perfect accuracy. Its findings are hard to verify because the data splits and external validation set are under-specified.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BUSI results may be inflated by image-level rather than patient-level data splitting; the paper never states that the 80/10/10 split separates patients despite BUSI having multiple images per patient.","rationale":"The reader's weakest assumption—patient-level leakage—is the same load-bearing concern I would identify. The paper never states that the 80/10/10 split is at the patient level, and Algorithm 1's 'stratified' almost certainly refers to class stratification, not patient stratification. Given that BUSI has 780 images from 600 patients, random image-level splitting will place images from the same patient in both training and test sets, which can inflate near-perfect metrics. This is the most direct threat to the central claim of 99.7% accuracy and perfect malignant recall. The external-validation inconsistency (95.2% on 100 images) is a separate but corroborating sign that the reported numbers are internally unreliable. I do not see the patient-leakage concern as resolved by anything in the paper: no patient-level split, no error bars, no code, and no test-set construction details are given. Therefore, the central claim is not reproducible as stated, and the reader's REJECT verdict stands. If the proposed patient-level re-run were performed and the numbers held, the verdict could be revisited, but the current manuscript does not provide the necessary evidence.","tokens_in":8638,"tokens_out":5776,"duration_ms":62430,"concrete_test":"Reproduce the BUSI experiment with a patient-level split: parse the patient identifier embedded in the BUSI image filenames (e.g., 'benign (n).png' maps to patient n), group images by patient, stratify patients by class, and assign 80/10/10 of patients to train/val/test. Retrain ResNet-18 with the exact preprocessing and augmentation described in Algorithm 1, then compare test accuracy and malignant recall with Table I. If accuracy drops by more than ~1 percentage point or recall falls below 1.00, the headline is inflated by patient-level leakage. As a secondary check, compute the maximum pairwise image similarity (e.g., normalized cross-correlation) between training and test images to detect near-duplicate leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—ResNet-18 at 99.7% accuracy with perfect malignant recall (Table I)—rests on the test split being clean. BUSI contains 780 images from 600 patients (Section II-A), so many patients contribute multiple images. The paper's only split description (Section II-E and Algorithm 1 step 1) says 'stratified 80/10/10', but the stratification variable is the class label, not the patient. If images from the same patient (or same lesion) appear in both training and test, the model has effectively seen near-identical views during training, inflating test accuracy and recall. No patient-level split is stated, no patient IDs are provided, and no code is released to check. This directly threatens the headline number. A separate arithmetic red flag: Section III-C reports 95.2% accuracy on '100 images' for external validation, which is impossible with integer errors (95/100 = 95%, 96/100 = 96%). This suggests the numerical reporting is not precise enough to rule out leakage artifacts. The concern is an omission in reporting, not an accusation of intent, but it is load-bearing because the paper's clinical-feasibility argument depends entirely on the near-perfect test numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates classical ML (SVM, KNN) and deep transfer-learning CNNs (AlexNet, VGG16, GoogLeNet, ResNet-18, EfficientNet-B0) for three-class breast ultrasound classification (normal/benign/malignant). The central empirical claim is that ResNet-18 reaches 99.7% accuracy with perfect recall for malignant lesions on a combined BUSI/BUS-BRA evaluation, with an external validation accuracy of 95.2%. The paper also compares deep-feature SVM/KNN, provides Grad-CAM interpretability, and proposes an algorithmic pipeline for clinical deployment.","tokens_in":8959,"tokens_out":4312,"duration_ms":46677,"significance":"If the reported results are reproducible and leakage-free, the paper would provide a strong benchmark: a standard transfer-learned ResNet-18 with high sensitivity on public breast ultrasound datasets, plus interpretability via Grad-CAM and a usable algorithmic formulation. The comparison between classical classifiers with deep features and full CNNs is useful, and the use of publicly available BUSI and BUS-BRA datasets is a strength. However, the significance is conditional on resolving serious reporting gaps: no patient-level split is documented, no confidence intervals are given, the external validation set is unnamed and arithmetically inconsistent, and the multi-dataset framing exceeds the actual experiments. The paper does not release code or patient identifiers, so the central claim is currently unverifiable.","major_comments":[{"comment":"BUSI contains 780 images from 600 patients (Section II-A). The split description (Section II-E and Algorithm 1, Step 1) states only a 'stratified 80/10/10' split, with class labels as the stratification variable, and does not state that all images from a given patient are kept in the same partition. If one patient contributes multiple images, near-duplicate views of the same lesion can appear in both training and test, inflating the 99.7% accuracy and perfect malignant recall in Table I. Please specify a patient-level split, report the number of patients per partition, and provide patient-clustered metrics. If such a split was used, state it explicitly; if not, rerun the experiments.","section":"II-A, II-E, Algorithm 1"},{"comment":"Table I reports point estimates only, with no confidence intervals, fold-level standard deviations, or statistical comparisons among models. Section II-E mentions both 5-fold cross-validation and 80/10/10 train-validation-test splits, but it is not stated whether Table I reports a single held-out test set, an average over folds, or a specific split. For a headline claim of 99.7% accuracy and recall 1.00, binomial or bootstrap confidence intervals are necessary to assess whether the differences among ResNet-18, GoogLeNet, and EfficientNet-B0 are meaningful.","section":"III-A, Table I"},{"comment":"The external validation reports '100 images' and 95.2% accuracy. On a single set of 100 examples, accuracy must be a multiple of 1% (95% or 96%), so 95.2% is arithmetically inconsistent unless the value is an average over folds/subsets or the total is not 100. The archive itself is also unnamed. Please identify the external dataset, give the exact number of images and patients, provide the confusion matrix, and reconcile the reported accuracy with the sample count. This matters because the external validation is the primary evidence for the generalization claim.","section":"III-C"},{"comment":"The abstract and contribution list announce a 'multi-dataset' study, but a later paragraph in Section II-A states that the reported results were 'derived from experiments on the BUSI and BUS-BRA datasets' and that BrEaST-Lesions USG, the Breast Ultrasound Archive, and the Hybrid dataset were used only for segmentation, augmentation, or external validation. Table I indeed covers only BUSI and BUS-BRA. Please either present per-dataset results for every dataset used or revise the framing so that 'multi-dataset' accurately describes the actual experiments.","section":"II-A and Abstract/Contributions"},{"comment":"Algorithm 1 selects the best model by maximizing malignant recall, AUC, and accuracy. The text does not clarify whether this selection is performed on the validation split or on the test split. If the test set is used for model selection, the reported superiority of ResNet-18 is selection bias, not an independent comparison. Please state the selection protocol explicitly and, if test data influenced the choice of architecture, either correct the claim or report validation-based selection results.","section":"Algorithm 1, Step 8"}],"minor_comments":[{"comment":"The sentence 'zoom up to 10' is incomplete; specify whether the zoom range is 10% or 10 pixels and how it is implemented.","section":"II-B"},{"comment":"The text refers to 'Algo.??' as an unresolved placeholder; Algorithm 1 is referenced before it is introduced. Please fix the cross-reference.","section":"II-E"},{"comment":"The paragraph beginning 'In this study, we indeed discussed five datasets...' reads as an inserted clarification rather than part of the Methods. Please integrate this information into the experimental design description and keep it consistent with the abstract.","section":"II-A"},{"comment":"Figures 1 and 2 are referenced but not included in the text I received. Ensure the final version contains the confusion matrix and ROC curves with labeled axes and legends.","section":"Figures"},{"comment":"The Limitations section should explicitly address the absence of patient-level split documentation and the lack of confidence intervals, as these directly affect the interpretation of the reported near-perfect results.","section":"IV-C"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to contain an inserted authorial clarification in Section II-A that partially contradicts the abstract's multi-dataset framing; in its current form it reads like an unpolished draft. The heavy self-citation pattern is not itself a technical flaw, but it may merit editorial attention. The main technical risk is patient-level leakage in the BUSI split and the unverifiable external validation set."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is not a new method—it's a standard transfer-learning bake-off on two public ultrasound datasets. The reported ResNet-18 numbers (99.7% accuracy, perfect malignant recall) are plausible given prior work, but the paper as written does not support them. I'd send it back for major revisions, or reject if the authors can't fix the reporting.\n\nWhat's genuinely useful: the paper gives a clear, step-by-step pipeline (preprocessing, feature extraction, model training, evaluation, Grad-CAM) and a sensible comparison of classical ML with and without deep features, CNNs of varying depth, and inference-time considerations. The emphasis on malignant recall as the selection criterion is clinically sane. If the numbers were reproducible, this would be a convenient benchmark for practitioners scoping CAD systems.\n\nThe soft spots are not minor. First, no patient-level split is stated despite BUSI having 780 images from 600 patients. Algorithm 1 says 'stratified 80/10/10', which sounds class-stratified, not patient-stratified. If images from the same patient leak across train and test, the headline accuracy is inflated. Second, the external validation claim of 95.2% on '100 images' is arithmetically impossible with whole-image counts. That kind of sloppiness makes me distrust every number in Table I. Third, the external archive is never named, so reproducibility is nil. Fourth, there are no confidence intervals or error bars. Fifth, several references (e.g., [12], [14], [15], [17], [18]) look like placeholder or unverifiable entries; that's a citation-hygiene problem that undermines the lit review.\n\nNone of this is fatal in the sense of a flawed idea—the methodology is standard and the framing is coherent. But the evidence for the 'state-of-the-art' claim is weak as presented. The paper adds no new technique, loss, or theoretical insight; its contribution is a benchmark number, and that number isn't verifiable.\n\nWho would get value? Applied researchers or clinicians wanting a quick survey and a template pipeline. But they should not rely on the absolute numbers without code and data. I'd bring it to a reading group as a cautionary example, and I'd be willing to referee a substantially revised version if the authors clarify the split, name the external set, fix the arithmetic, and release code. As it stands, I would not cite it.\n\nRecommendation: desk reject in current form, or conditional reject with explicit invitation to resubmit after addressing leakage and reporting. Either way, don't take the 99.7% at face value.","headline":"A routine transfer-learning benchmark whose headline numbers are undercut by reporting gaps: no patient-level split stated, an impossible 95.2% on 100 images, and an unnamed external set. Reject as is, but not a waste of referee time.","tokens_in":9383,"tokens_out":3041,"would_cite":false,"duration_ms":29970,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ResNet-18 classifies breast ultrasound lesions at 99.7% accuracy with no missed malignancies in this multi-dataset study.","keywords":["breast ultrasound","transfer learning","ResNet-18","deep features","Grad-CAM","computer-aided diagnosis","BUSI","BUS-BRA"],"falsifier":"Run the same ResNet-18 preprocessing and training pipeline on BUSI using a strict patient-level split (ensuring no patient appears in both training and test) and report the test accuracy and malignant recall; if accuracy drops substantially below 99.7%, the headline result depends on image-level leakage rather than true generalization.","tokens_in":8610,"feed_emoji":"🩺","tokens_out":2495,"duration_ms":26413,"temperature":0.7,"pith_summary":"The paper tries to establish that transfer learning with off-the-shelf CNN architectures, especially ResNet-18, can classify breast ultrasound images into normal, benign, and malignant categories at near-perfect accuracy. Using the BUSI and BUS-BRA datasets, it reports 99.7% accuracy and 100% recall for malignant cases, and shows that deep features extracted from pretrained CNNs lift classical classifiers (SVM, KNN) close to CNN performance. The authors argue that this combination of high sensitivity, interpretability via Grad-CAM, and external validation supports practical AI-assisted breast cancer screening, particularly in resource-limited settings. The study is an application benchmark rather than a new method, but its headline result would matter clinically if it holds under rigorous patient-level evaluation.","feed_headline":"ResNet-18 hits 99.7% accuracy on breast ultrasound","feed_subtitle":"Transfer learning plus Grad-CAM maps brings benign-versus-malignant classification to near-perfect levels on public datasets, with no missed","key_machinery":"The central mechanism is transfer learning: pretrained ImageNet CNNs (ResNet-18, EfficientNet-B0, GoogLeNet, AlexNet, VGG16) are fine-tuned on breast ultrasound images after a preprocessing pipeline of grayscale normalization, median-filter speckle reduction, ROI cropping using masks, and data augmentation. Deep features extracted from the penultimate layers are also fed into SVM and KNN classifiers, showing that classical models gain substantially from learned representations. Grad-CAM is used to generate class-activation overlays that localize the image regions driving each prediction.","core_discovery":"On the paper's own terms, the central discovery is that a standard ResNet-18 fine-tuned on breast ultrasound images achieves 99.7% classification accuracy and perfect sensitivity for malignant lesions on a test split of BUSI and BUS-BRA data. Deep-feature SVM reaches 99.3% accuracy, and EfficientNet-B0 achieves 99.5% with lower latency. External validation on an archive of 100 unseen images yields 95.2% accuracy, which the authors interpret as strong generalization with expected domain shift. The paper also demonstrates that Grad-CAM heatmaps align with diagnostically relevant regions, supporting the case that such models are interpretable enough for clinical second-reader roles.","pith_inferences":["The reported 99.7% accuracy likely depends on the exact train/test split construction; since BUSI has multiple images per patient, a patient-level split could lower the numbers substantially, and that remains untested here.","The paper's preprocessing includes ROI cropping from segmentation masks, which are not available in real-time clinical scans; an end-to-end detection framework would be needed to translate these results into practice.","Perfect malignant recall on a small test set is not strong evidence of zero false negatives at scale; testing on larger, more diverse cohorts with class imbalance would give a better estimate of the real-world miss rate.","If the external validation set was sampled from a single archive, the 95.2% accuracy may not generalize across scanner vendors or patient populations; multi-center prospective data would be the next test."],"forward_implications":["If the 99.7% accuracy and perfect malignant recall hold under strict patient-level splits, a transfer-learned ResNet-18 could serve as a reliable second-reader tool in breast ultrasound screening workflows.","Deep features extracted from pretrained CNNs can make classical classifiers competitive with dedicated deep models, enabling deployment on mid-tier hardware in low-resource clinics.","Grad-CAM visualizations provide a practical interpretability layer that may increase radiologist trust and help flag ambiguous cases for manual review.","External validation at 95.2% accuracy quantifies a real performance drop under domain shift, indicating that multi-center training or adaptation is needed before clinical deployment.","The paper's model-selection rule, prioritizing malignant recall over overall accuracy, aligns the evaluation metric with clinical risk: missing a cancer is costlier than a false alarm."],"supporting_citations":[{"why":"Supplies the BUSI dataset of 780 ultrasound images with masks and three-class labels, the primary training and evaluation data.","marker":"[24]"},{"why":"Supplies the BUS-BRA dataset with BI-RADS labels and biopsy ground truth, used as the second benchmark.","marker":"[25]"},{"why":"Defines EfficientNet-B0, the lightweight architecture compared for accuracy-latency trade-offs.","marker":"[26]"},{"why":"Provides support that deep-feature fusion lifts classical classifiers close to deep learning performance on breast ultrasound.","marker":"[27]"},{"why":"Establishes the prior baseline that CNNs reach 92% sensitivity in breast ultrasound, which this paper aims to exceed.","marker":"[12]"},{"why":"Reports a hybrid CNN-feature plus classical classifier approach with 94% accuracy, a direct competitor this study benchmarks against.","marker":"[14]"},{"why":"Shows the importance of multi-center external validation for deep learning in breast ultrasound, motivating the paper's external test.","marker":"[19]"}],"fun_headline_variants":["AI breast ultrasound hits 99.7% accuracy","ResNet-18 bests breast cancer detection on ultrasound","Interpretable AI catches breast cancer on ultrasound","Deep transfer learning scores 99.7% on breast ultrasound"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The 80/10/10 train-validation-test split is not explicitly stated to be at the patient level even though BUSI contains multiple images per patient, so images from the same patient may appear in both training and test sets, which would inflate the reported accuracy and perfect recall.","fun_headline_variants_meta":{"raw":{"variants":["AI breast ultrasound hits 99.7% accuracy","ResNet-18 bests breast cancer detection on ultrasound","Interpretable AI catches breast cancer on ultrasound","Deep transfer learning scores 99.7% on breast ultrasound"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00014,"raw_usage":{"total_tokens":990,"prompt_tokens":729,"completion_tokens":261,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":197}},"tokens_in":473,"tokens_out":261,"duration_ms":3396,"temperature":1.0,"reasoning_tokens":197,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:40:17.117333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same ResNet-18 preprocessing and training pipeline on BUSI using a strict patient-level split (ensuring no patient appears in both training and test) and report the test accuracy and malignant recall; if accuracy drops substantially below 99.7%, the headline result depends on image-level leakage rather than true generalization.","supporting_citations":[{"cited_title":"BUS- BRA: A breast ultrasound dataset for assessing computer-aided diag- nosis systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the BUS-BRA dataset with BI-RADS labels and biopsy ground truth, used as the second benchmark."},{"cited_title":"Efficientnet: Rethinking model scaling for convolutional neural networks,","cited_arxiv_id":null,"evidence_quote":"Defines EfficientNet-B0, the lightweight architecture compared for accuracy-latency trade-offs."},{"cited_title":"Breast cancer classification from ultrasound images using probability-based optimal deep learning feature fusion,","cited_arxiv_id":null,"evidence_quote":"Provides support that deep-feature fusion lifts classical classifiers close to deep learning performance on breast ultrasound."},{"cited_title":"Deep learning for breast cancer detection in ultrasound im- ages: A comprehensive review,","cited_arxiv_id":null,"evidence_quote":"Establishes the prior baseline that CNNs reach 92% sensitivity in breast ultrasound, which this paper aims to exceed."},{"cited_title":"Hybrid models for breast cancer detection: Combining CNNs and traditional ML approaches,","cited_arxiv_id":null,"evidence_quote":"Reports a hybrid CNN-feature plus classical classifier approach with 94% accuracy, a direct competitor this study benchmarks against."},{"cited_title":"Multicenter evaluation of deep learning for breast ultrasound diagnosis,","cited_arxiv_id":null,"evidence_quote":"Shows the importance of multi-center external validation for deep learning in breast ultrasound, motivating the paper's external test."}],"review_version":1}