{"id":"179a4aba-a749-47a6-8835-bf46b66eb342","arxiv_id":"2506.07228","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"VGG16 with Grad-CAM and Grad-CAM++ classifies three brain tumor types from Bangladeshi MRI scans at 99.17% test accuracy, best among eight transfer learning models.","lead":"This paper benchmarks eight pre-trained image models, plus two heatmap explanation methods, for classifying brain tumor MRI scans from a public Bangladesh dataset. An interpretable VGG16 model reaches about 99% test accuracy, which is the kind of result hospitals in low-resource settings would want to see validated more rigorously before use.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 99.17% accuracy rests on an unreproducible, non-stratified split; test counts deviate from dataset proportions and Table 1 metrics disagree with the confusion matrix.","rationale":"The paper's central claim has two parts: a quantitative accuracy result and a qualitative XAI contribution. I focus on the accuracy result because the headline number (99.17%) and the comparison across eight models are the paper's primary evidence. The accuracy is internally consistent with Figure 4(a), which is a point in the authors' favor. However, the result is not reproducible from the paper: no seed, no code, and no split artifacts are given, and the reported test-set class counts deviate from the dataset's stated proportions. The additional mismatch between Table 1's per-class metrics and the confusion matrix indicates that at least some reported numbers are erroneous. These issues are concrete and checkable; if the split cannot be reproduced or is non-representative, the claimed accuracy and model ranking lose their evidentiary value. The XAI section also lacks quantitative validation, but that weakness affects the novelty interpretation rather than the numerical claim. The reader correctly identified split and dataset issues as the weakest assumption; I agree partially and sharpen the concern to reproducibility and internal consistency. A conditional verdict remains appropriate: the study is a routine benchmark with public data, so the key numbers can be verified with modest effort.","tokens_in":8474,"tokens_out":10105,"duration_ms":109659,"concrete_test":"Download the Mendeley 'Brain Cancer - MRI Dataset,' verify the class counts, and reproduce the described pipeline with a fixed-seed stratified 80:10:10 split. Then (1) compare the resulting test-set class counts to the 209/217/180 implied by Figure 4(a); (2) recompute VGG16 per-class metrics from the confusion matrix and compare them to Table 1. If the counts cannot be matched or the accuracy across 5 seeds varies by more than 1 percentage point, the reported 99.17% is an artifact of the particular split rather than a stable property of the model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—VGG16's 99.17% accuracy and its status as best model—rests on a single, unreported split of the Mendeley dataset. Section 3.3 specifies an 80:10:10 split but gives no seed, no stratification, and no split artifacts. The VGG16 confusion matrix in Figure 4(a) implies test counts of 209 glioma, 217 meningioma, and 180 tumor (total 606). Given the class proportions stated in Section 3.1 (2004/2004/2048 out of 6056), a simple random 10% split would yield expectations of about 200.5, 200.5, and 205.0; the observed tumor count of 180 is roughly 2.1 standard deviations below expectation, suggesting the split is not a typical random draw. More importantly, the per-class precision/recall/F1 values in Table 1 cannot be derived from Figure 4(a): for VGG16, the matrix gives Glioma precision 207/207 = 1.00 and recall 207/209 = 0.99, whereas Table 1 lists 0.98/0.98/0.98. This internal inconsistency means the paper's detailed performance table is unreliable, and without a released split or seed the accuracy claim cannot be independently checked. If the split is not representative or reproducible, the claim that VGG16 is the best model and achieves 99.17% may not generalize beyond this particular test set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports a transfer-learning study for classifying three brain-tumor classes (glioma, meningioma, tumor) in a publicly available MRI dataset sourced from Bangladeshi hospitals. Eight pretrained CNN architectures are fine-tuned, with VGG16 reported as the best at 99.17% accuracy. Grad-CAM and Grad-CAM++ are used to produce heatmaps intended to show which image regions drive the classifications. The paper's central claims are that VGG16 outperforms the other models, that the localized Bangladeshi dataset provides a region-specific contribution, and that the dual-XAI approach improves clinical interpretability.","tokens_in":8708,"tokens_out":3636,"duration_ms":37479,"significance":"If the results hold, the study would provide a useful reference point for applying standard transfer-learning pipelines to a regional MRI dataset, with a plausible accuracy level in line with prior work on the same task. The attempted use of two XAI methods adds transparency, although the evaluation of the explanations is qualitative. The main contribution is the use of a Bangladeshi dataset and the comparative benchmarking, which is incremental but potentially useful for practitioners in low-resource settings. The paper would be strengthened by making the split reproducible and by reconciling the reported per-class metrics with the confusion matrices.","major_comments":[{"comment":"Table 1 and Figure 4(a) are internally inconsistent for the central models. For VGG16, the confusion matrix gives Glioma precision 207/207 = 1.00 and recall 207/209 = 0.99, yet Table 1 lists 0.98 precision, 0.98 recall, and 0.98 F1. Similar discrepancies appear for other classes and for VGG19 (e.g., Figure 4(b) implies Meningioma recall 207/217 = 0.95, not 0.97 as printed). Because Table 1 is the primary evidence for the comparative claim that VGG16 is best, these inconsistencies undermine the detailed performance comparison and must be corrected or explained.","section":"§4, Table 1 vs Figure 4(a)"},{"comment":"The train/validation/test split is not reproducible. Section 3.3 specifies an 80:10:10 ratio but gives no random seed and does not mention stratification. The test counts implied by Figure 4(a) (209 glioma, 217 meningioma, 180 tumor) deviate markedly from what a simple random 10% split would be expected to yield under the dataset proportions of Section 3.1 (roughly 200, 200, and 205): the tumor count of 180 is approximately 2.1 standard deviations below expectation. Without a released split, a seed, or stratification details, the central 99.17% accuracy claim cannot be independently verified, and the possibility that the test set is unrepresentative remains open.","section":"§3.3, Dataset Splitting"},{"comment":"The claim that Grad-CAM and Grad-CAM++ heatmaps 'validate' the model's predictions is not supported by any quantitative or structured evaluation. Section 5 states that overlapping highlighted regions 'validates their importance,' but no overlap metric, expert reader study, or comparison against tumor segmentations is provided. Since explainability is a stated contribution of the paper, the current qualitative demonstration is insufficient to support the claim that the XAI component enhances 'stability' or clinical trust.","section":"§5, Explainable AI evaluation"},{"comment":"Key hyperparameters and experimental details are under-reported. Section 4 mentions 30 epochs, a learning-rate range from 1e-3 to 1e-5, and the Adam optimizer, but does not report the exact learning rate, batch size, dropout values, or early-stopping criteria for each model. No code or configuration files are provided. These omissions hamper replication, especially given that the split itself is not reproducible.","section":"§4, Experimental reproducibility"}],"minor_comments":[{"comment":"The title contains a typo ('A S TUDY' should be 'A STUDY') and the keyword list includes 'VGGNNet', which appears to be a misspelling of 'VGGNet'.","section":"Title and keywords"},{"comment":"The claim that the dataset 'was collected from various hospitals across Bangladesh' is taken from the Mendeley record [13]; the manuscript provides no additional verification of the provenance or of label quality. This is worth stating as a limitation.","section":"§3.1, Dataset provenance"},{"comment":"The phrase 'This division guarantees a balanced training methodology' is imprecise: an 80:10:10 split does not by itself guarantee class balance, and the paper does not report using stratified sampling.","section":"§3.3, Wording"},{"comment":"The formatting of Table 1 is ambiguous: the accuracy column appears once but the per-class precision/recall/F1 rows are nested under each model. This layout makes it easy to misread the table and should be reformatted for clarity.","section":"§4, Table 1 layout"},{"comment":"The Grad-CAM++ equation uses the same symbol α_c^k as Grad-CAM but with a different weighting scheme; the notation does not make clear that the two equations define different quantities. A brief sentence distinguishing the two weight definitions would improve readability.","section":"§5, Equations"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a preprint of a paper already published at IEEE STI 2024 (DOI 10.1109/STI64222.2024.10951092). The referee report addresses the arXiv manuscript as submitted. The internal inconsistency between Table 1 and the confusion matrices is the main load-bearing issue; it is correctable in a revision. The lack of an exact split is a reproducibility concern that should also be addressed. If the authors provide the corrected metrics and split details, I would be willing to reconsider."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things up front. First, this is a standard transfer-learning benchmark on the Mendeley Bangladesh Brain Cancer MRI Dataset, which no prior cited work used. Second, the headline VGG16 accuracy of 99.17% is internally supported: the confusion matrix in Figure 4(a) gives exactly (207+215+179)/606. That is real, and I credit the paper for publishing the matrix.\n\nWhat is genuinely new is the localized benchmark itself: eight ImageNet-initialized TL models fine-tuned on a Bangladeshi hospital dataset, with Grad-CAM and Grad-CAM++ heatmaps. It is not a methodological advance, and the authors do not pretend it is. The paper is clearly written, the architecture description is adequate, and the comparison with related work in Table 2 is useful.\n\nNow the soft spots. The per-class precision/recall/F1 in Table 1 cannot be derived from the confusion matrix in Figure 4(a). For VGG16 Glioma, the matrix gives precision 1.00 and recall 0.99, but Table 1 says 0.98/0.98/0.98. That discrepancy is mechanical, not fatal, but it means the detailed performance table is unreliable as printed. Second, the 80:10:10 split is reported without a seed or stratification. The test counts (209/217/180) deviate noticeably from what a random split of the stated class proportions would produce, so the 99.17% figure may not generalize to a different draw. Third, no code, configuration, or split artifacts are released, so the result is not reproducible from the paper alone. Fourth, the XAI section claims the heatmaps \"validate\" predictions and that XAI improves \"stability,\" but no quantitative evaluation backs these claims; the visualizations are anecdotal.\n\nThese are addressable, and they do not contradict the order-of-magnitude conclusion that VGG16 performs very well on this dataset. The central contribution is a data point, not a proof.\n\nThe paper is for readers working on low-resource medical imaging benchmarks who want a reference accuracy for this specific dataset. If I were editing, I would send this to serious peer review: the core numerical claim is checkable and probably correct, but the authors should be asked to fix Table 1, release the split or at least report a seed and stratification, and tone down the XAI claims. I would not cite it in my own work until those fixes are made, but it deserves referee time.","headline":"A routine but internally consistent transfer-learning benchmark on a new Bangladeshi MRI dataset; the headline VGG16 accuracy checks out against the confusion matrix, yet unreported splitting and table inconsistencies undercut reproducibility.","tokens_in":9323,"tokens_out":1216,"would_cite":false,"duration_ms":15529,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fine-tuned VGG16 can classify Bangladeshi brain MRI scans into glioma, meningioma, and tumor classes with 99.17% accuracy, and that Grad-CAM and Grad-CAM++ heatmaps localize the tumor regions driving the predictions.","keywords":["brain tumor classification","transfer learning","VGG16","explainable AI","Grad-CAM","Grad-CAM++","Bangladeshi MRI data"],"falsifier":"Re-run the experiment on the same dataset but with a fixed random seed, stratified by class, and with a sample of labels checked by a radiologist; if VGG16's accuracy falls materially below 99.17% or per-class F1 varies widely across seeds, the published figure reflects split luck or label leakage rather than generalization. A second check: test the same fine-tuned VGG16 on MRI scans from a different Bangladeshi hospital not represented in the dataset and see whether accuracy stays near 99%.","tokens_in":8184,"feed_emoji":"🧠","tokens_out":6479,"duration_ms":60365,"temperature":0.7,"pith_summary":"This paper tries to establish that a transfer learning pipeline built on ImageNet-pretrained VGG16 can accurately classify brain tumors from MRI data collected in Bangladesh, and that the model's decisions can be made explainable. The paper reports 99.17% accuracy on the 6,056-image, three-class Bangladeshi dataset, beating seven other pretrained models, and it argues that Grad-CAM and Grad-CAM++ heatmaps consistently highlight tumor areas, supporting clinician trust. This matters because automated diagnosis in resource-limited settings needs both high accuracy and transparency to be usable in practice. The contribution is presented as an extension: localized data and dual explainability rather than a new architecture.","feed_headline":"VGG16 hits 99.17% accuracy on Bangladeshi brain MRI scans","feed_subtitle":"Fine-tuned VGG16 plus Grad-CAM heatmaps localizes tumor regions, so clinicians can see what drives each diagnosis.","key_machinery":"The central mechanism is the transfer learning architecture: ImageNet-pretrained convolutional feature extractors (VGG16, VGG19, DenseNet, InceptionResNetV2, MobileNetV2, Xception, ResNet50V2, InceptionV3) topped with flatten, dense, and dropout layers and fine-tuned on resized, normalized, and augmented 128x128 MRI images. Training uses sparse categorical cross-entropy with Adam optimization over roughly 30 epochs. The explainability machinery is Grad-CAM, which weights feature maps by globally averaged gradients to produce a class-discriminative heatmap, and Grad-CAM++, which adds second-order gradient terms for finer localization; these heatmaps are the object that links the high accuracy claim to the clinical-transparency claim.","core_discovery":"On its own terms, the paper's central claim is empirical: on the Bangladesh Brain Cancer MRI Dataset, fine-tuned VGG16 is the best of the eight transfer learning models tested, with 99.17% disease accuracy and per-class F1 scores at or near 0.98 for glioma, meningioma, and tumor. VGG19 and DenseNet follow at 98% accuracy. The paper further claims that Grad-CAM and Grad-CAM++ visualizations show the models fixating on tumor cores, with Grad-CAM++ giving finer localization in anatomically complex cases such as glioma, and it treats this overlap as evidence that predictions rest on clinically meaningful regions. The proposed contribution is therefore not a new network but a combination of a publicly available local dataset, a fine-tuning recipe, and dual XAI display.","pith_inferences":["Because the paper reports no random seed, no stratified split, and test-set class counts (209/217/180 in Figure 4(a)) that deviate from the dataset's proportional split, the headline accuracy is likely to shift under repeated random splits; a seeded, stratified replication would give a more honest estimate.","The dataset's images come from a single public collection, so 99.17% accuracy says little about performance on scans from different machines or protocols; an external validation on a second Bangladeshi hospital's MRI data would test domain transfer.","If label noise exists (unverified labels are a plausible source), the high agreement between Grad-CAM and Grad-CAM++ mainly shows the model learned the dataset's cues, not that those cues are clinically correct; expert label verification would separate these two readings.","A practical extension: the same pipeline could be reshaped into a screening tool that flags suspicious scans for radiologist review, with the heatmap as the human-checkable evidence."],"forward_implications":["If the 99.17% figure holds, fine-tuned VGG16 can be deployed as a high-accuracy first-pass classifier on Bangladeshi hospital MRI scans, with per-class F1 scores around 0.98.","The Grad-CAM/Grad-CAM++ heatmaps provide an inspectable rationale for each prediction, which is the missing piece for clinicians to trust automated reads in low-resource settings.","The same fine-tuning recipe, pretrained weights plus custom dense layers plus augmentation, can be transferred to other neuroimaging classification tasks where local data is scarce.","The comparison table implies that accuracy parity with earlier Bangladeshi studies is achievable without a custom architecture, so the practical differentiator is local validation and explainability rather than model design."],"supporting_citations":[{"why":"Supplies the Bangladesh Brain Cancer MRI Dataset (6,056 images, three classes) on which all reported accuracies are measured.","marker":"[13]"},{"why":"Defines Grad-CAM, the gradient-weighted localization technique used to produce class-discriminative heatmaps.","marker":"[16]"},{"why":"Defines Grad-CAM++, whose second-order gradient weighting provides the finer localization maps.","marker":"[18]"},{"why":"Provides the prior CNN-based brain tumor detection result (97.87%) on BRATS that the paper benchmarks against.","marker":"[2]"},{"why":"Supplies the earlier explainable cost-sensitive deep network result (99.33%) that motivates pairing XAI with class-imbalance handling.","marker":"[3]"},{"why":"Reports 99.60% accuracy with MobileNet, InceptionV3, and DenseNet121 transfer learning, the main accuracy target this paper compares itself to.","marker":"[5]"},{"why":"Presents VGG-SCNet, a VGG-based stacked classifier with 99.2% F1, the closest architectural baseline for the VGG16 result.","marker":"[12]"}],"fun_headline_variants":["VGG16 tops 99% accuracy on Bangladeshi brain MRI","Grad-CAM explains VGG16's 99% brain tumor calls","Bangladeshi MRIs classified with 99% accuracy and XAI","Fine-tuned VGG16 hits 99.17% on brain tumors with XAI","Deep learning reads Bangladeshi brain scans at 99%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 'Brain Cancer - MRI Dataset' is correctly labeled and representative of Bangladeshi clinical MRI practice, and that the 80:10:10 split is a random, unbiased sample; the paper verifies neither the labels nor the split's class balance.","fun_headline_variants_meta":{"raw":{"variants":["VGG16 tops 99% accuracy on Bangladeshi brain MRI","Grad-CAM explains VGG16's 99% brain tumor calls","Bangladeshi MRIs classified with 99% accuracy and XAI","Fine-tuned VGG16 hits 99.17% on brain tumors with XAI","Deep learning reads Bangladeshi brain scans at 99%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1441,"prompt_tokens":947,"completion_tokens":494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":397}},"tokens_in":563,"tokens_out":494,"duration_ms":4626,"temperature":1.0,"reasoning_tokens":397,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:39:03.965923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the experiment on the same dataset but with a fixed random seed, stratified by class, and with a sample of labels checked by a radiologist; if VGG16's accuracy falls materially below 99.17% or per-class F1 varies widely across seeds, the published figure reflects split luck or label leakage rather than generalization. A second check: test the same fine-tuned VGG16 on MRI scans from a different Bangladeshi hospital not represented in the dataset and see whether accuracy stays near 99%.","supporting_citations":[{"cited_title":"Brain cancer - mri dataset, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the Bangladesh Brain Cancer MRI Dataset (6,056 images, three classes) on which all reported accuracies are measured."},{"cited_title":"Grad-cam: visual explanations from deep networks via gradient-based localization","cited_arxiv_id":null,"evidence_quote":"Defines Grad-CAM, the gradient-weighted localization technique used to produce class-discriminative heatmaps."},{"cited_title":"Grad-cam++: General- ized gradient-based visual explanations for deep convolutional networks","cited_arxiv_id":null,"evidence_quote":"Defines Grad-CAM++, whose second-order gradient weighting provides the finer localization maps."},{"cited_title":"Brain tumor detection using convolutional neural network","cited_arxiv_id":null,"evidence_quote":"Provides the prior CNN-based brain tumor detection result (97.87%) on BRATS that the paper benchmarks against."},{"cited_title":"Explainable Cost-Sensitive Deep Neural Networks for Brain Tumor Detection from Brain MRI Images considering Data Imbalance","cited_arxiv_id":"2308.00608","evidence_quote":"Supplies the earlier explainable cost-sensitive deep network result (99.33%) that motivates pairing XAI with class-imbalance handling."},{"cited_title":"Transfer learning architectures with fine-tuning for brain tumor classification using magnetic resonance imaging.Healthcare Analytics, 4:100270, 2023","cited_arxiv_id":null,"evidence_quote":"Reports 99.60% accuracy with MobileNet, InceptionV3, and DenseNet121 transfer learning, the main accuracy target this paper compares itself to."},{"cited_title":"Vgg-scnet: A vgg net-based deep learning framework for brain tumor detection on mri images","cited_arxiv_id":null,"evidence_quote":"Presents VGG-SCNet, a VGG-based stacked classifier with 99.2% F1, the closest architectural baseline for the VGG16 result."}],"review_version":1}