{"id":"0bbb571f-699e-4d1a-955a-51cd804b7bde","arxiv_id":"2411.15596","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-layer CNN matches pretrained ResNet18 and VGG16 accuracy on brain tumor MRI classification while running several times faster.","lead":"Researchers compared a tiny custom CNN with two pretrained deep networks for classifying brain tumors from MRI images. The small model came within a fraction of a percent of the big networks' accuracy while being much faster and cheaper, suggesting that simple models can be enough for some medical imaging tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The custom-CNN accuracies in Table 7 are test-set selected maxima from a four-way learning-rate sweep; without a validation split or repeated runs, the headline margins are not unbiased estimates.","rationale":"The reader's weakest assumption correctly identifies the test-set-based learning-rate selection as the main threat. My stress-test concurs: Section 3.4's 'best-performing learning rate' is selected on Tables 3-6 from test-set accuracy, and Table 7 reuses those exact maxima, so the headline custom-CNN numbers are selected rather than unbiased estimates. This is the single most load-bearing issue because the entire claim of 'competitive performance' is quantified by those accuracies. However, the concern does not force rejection: the gaps are small, all models are near ceiling on these datasets, and the qualitative conclusion (a simple CNN is within about one point of pretrained models) is plausible and likely robust to a cleaner protocol. The paper also has independent support where it reports concrete parameter counts, inference times, confusion matrices, and a fixed seed, though it lacks code and repeated runs. Since the reader already returned CONDITIONAL and my concern is the same one, no verdict change is needed; the appropriate action is to require the nested-split rerun before accepting the headline numbers.","tokens_in":11601,"tokens_out":3900,"duration_ms":36877,"concrete_test":"Re-run the full comparison using an 80/10/10 nested split (or 5-fold cross-validation) for all three datasets. On the validation split, sweep the same four learning rates and pick the best; then evaluate the chosen model once on the held-out test split. Report mean and standard deviation over folds or over at least 5 random seeds, and report the four sweep accuracies for ResNet18 and VGG16 as well as for the custom CNNs. If the validation-selected test accuracies remain within 0.5 percentage points of Table 7 and the gaps to ResNet18/VGG16 keep the same sign, the central claim is supported; if the reported numbers drop materially or the ordering flips, the comparison is an artifact of test-set selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 splits data only into train/test (80/20, seed 42). Section 3.4 then runs a hyperparameter sweep over four learning rates and selects the rate with the highest test accuracy ('best-performing learning rate') for full fine-tuning. Tables 3-5 show the test accuracies; Table 6 selects the maximum (98.67%, 99.62%, 98.09%); Table 7 reports exactly these selected values as the custom-CNN results. Therefore the key comparison against ResNet18 and VGG16 is built on maxima chosen on the same test set that is later used for evaluation. This inflates the reported custom-CNN numbers and, more importantly, leaves the comparison without an unbiased estimate of accuracy; with 600/1311 test samples and four learning rates, the selection bias is of unknown magnitude but acts exactly in the direction of the reported headline values. No repeated seeds or error bars are reported, so it is unclear whether the observed margins are real or noise. The mislabeled '0-shot' random-weight baseline and absent code are secondary issues; the load-bearing flaw is the test-set selection protocol. This is a fixable methodological issue rather than a refutation: the qualitative claim that a small CNN is within a few points of large pretrained models may survive a cleaner evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two small custom CNNs (BTBCNN for binary classification, BTMCNN for multi-class classification) against pretrained ResNet18 and VGG16 on two brain tumor MRI datasets. The authors report that the custom models achieve accuracies of 98.67% on Br35H, 99.62% on the binary Brain Tumor MRI dataset, and 98.09% on the multi-class Brain Tumor MRI dataset, within about one percentage point of the pretrained models, while using far fewer parameters and lower inference times. The paper also presents few-shot learning experiments (5 to 80 shots per class) and a condition labeled '0-shot.' The central claim is that simple custom CNN architectures can be competitive with large pretrained models for this task at a substantially lower computational cost.","tokens_in":11744,"tokens_out":4373,"duration_ms":37748,"significance":"If the reported comparison were obtained from a sound evaluation protocol, the paper would provide a useful data point for resource-constrained medical imaging: it shows a small, task-specific CNN can approach the accuracy of much larger pretrained models on public brain tumor MRI benchmarks. Strengths include the detailed architecture descriptions, explicit parameter counts, and measured inference times. However, the core accuracy comparison is currently undermined by a test-set-based hyperparameter selection protocol, the absence of any variance estimates, and a mislabeled 'zero-shot' baseline. These issues are fixable and do not refute the underlying research question, but they must be corrected before the central claim can be considered reliable.","major_comments":[{"comment":"The learning-rate sweep is evaluated on the test set rather than on a held-out validation set. Section 3.2 splits the data only into training and testing sets, and Section 3.4 states that the 'best-performing learning rate' is selected from sweep results on the test accuracy shown in Tables 3-5. Table 7 then reports those same selected test accuracies as the final results for the custom CNNs. This post-hoc selection on the evaluation set means the reported custom-CNN accuracies are maxima of a four-way sweep, not unbiased estimates; with test sizes of 600 and 1311 samples, the selection bias is unknown but acts in the direction of the headline values. The comparison against ResNet18 and VGG16 is therefore not based on a fair or statistically clean protocol. Please redo the evaluation with a validation split (or nested cross-validation) for hyperparameter selection and a truly held-out test set for final evaluation, and apply the same protocol to all models.","section":"Section 3.4, Tables 3-6, Table 7"},{"comment":"The condition labeled '0-shot' is not zero-shot learning in the accepted sense. The custom CNNs are randomly initialized and are evaluated without any training, as the text itself attributes the result to the model's 'initial random weights' in Section 4.3. Zero-shot learning normally refers to generalization to unseen classes using a model with prior knowledge, such as a pretrained embedding or language model. Calling this condition '0-shot' is misleading and inflates the apparent contribution of the few-shot experiments. Please rename this baseline to 'random-weight baseline' or 'untrained baseline,' or implement an actual zero-shot protocol if the claim is to be retained.","section":"Section 4.3, Table 9"},{"comment":"The summary hyperparameter sweep table is internally inconsistent with Tables 3-5. For Br35H, Table 3 reports best accuracy 98.67% at LR=0.0005 and 97.00% at LR=0.0001, but Table 6 lists 98.67% under the LR=0.0001 column and 97.00% under the LR=0.0005 column. For the binary Brain Tumor MRI dataset, Table 4 reports 99.56% at LR=0.0001 and 99.54% at LR=0.0005, while Table 6 shows 99.54% for both of those columns and omits 99.56%. This makes it impossible to verify from the summary table which learning rate was actually selected and directly contradicts the detailed sweep tables. Please correct Table 6 so that its column order and values match Tables 3-5.","section":"Table 6"},{"comment":"All reported results are single runs with no variance estimates. Given the near-ceiling accuracies and the small reported gaps between models (for example, 98.67% vs. 99.33% on Br35H), single-run differences are within the range of random seed variation and do not reliably support the claim that the custom CNN is 'competitive' with ResNet18 and VGG16. Even after fixing the validation protocol, the paper should report mean and standard deviation over multiple random seeds (for example, 5-10 runs) and, ideally, a paired statistical comparison between the custom CNN and each pretrained model. Without such estimates, the central comparative claim is not statistically grounded.","section":"Section 4.2, Table 7"}],"minor_comments":[{"comment":"The preprocessing step that converts all images to grayscale removes color information that may be present in the original MRI scans; please justify this choice or provide an ablation showing that grayscale conversion does not harm performance.","section":"Section 3.2"},{"comment":"The inference-time table does not state the hardware, software framework, or batch size used for the measurements (beyond a batch size of 128), making the times difficult to interpret or reproduce.","section":"Table 8"},{"comment":"Several references are incomplete or inconsistent; for example, reference [9] lacks a URL, and the author name 'SanaUllah' in the text does not match 'Sana Ullah Khan' in the bibliography.","section":"References"},{"comment":"The phrase 'bias observed in 0-shot results' is vague; since the untrained baseline has random weights, the observed accuracy is simply a random-chance or initialization-dependent value, not a 'bias' in the statistical sense.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a workshop paper (IEEE ICCIT 2024) and is being considered for a more rigorous venue. The main methodological problem—test-set-based hyperparameter selection—must be fixed before the central claim can be accepted. The paper's novelty is moderate, so the decision should hinge on whether the authors can provide a clean evaluation with a proper validation split, repeated runs, and corrected few-shot terminology. If the authors cannot or will not redo the experiments, the reported accuracies cannot be taken at face value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible but methodologically under-powered comparison showing a small custom CNN can come within a fraction of a percent of ResNet18 and VGG16 on two public brain-tumor MRI datasets. The qualitative conclusion is probably right, but the headline numbers are not unbiased estimates: the authors selected the best of four learning rates using the test set itself, then reported those selected accuracies in the main comparison table. That's the load-bearing flaw.\n\nWhat's new and good: the specific architectures (two-layer BTBCNN, three-layer BTMCNN) are simple and described in enough detail to reimplement. The efficiency numbers—parameter counts, training time, per-batch inference time—are useful and directly support the resource-constrained angle. Comparing against both a binary and a four-class version of the same dataset is a reasonable design. The authors also honestly state limitations about dataset diversity and model simplicity. The few-shot experiment is a routine extension and they don't overclaim it.\n\nSoft spots, in order of severity. First, the learning-rate sweep is done on the test split (Section 3.4, Tables 3–6), and Table 7 reports exactly the selected maxima. With 600 test samples for Br35H and 1311 for the MRI dataset, and four learning rates, that selection can inflate accuracy by an unknown but likely nontrivial amount. No validation split, no repeated runs, no error bars. Second, the '0-shot' baseline is just evaluating randomly initialized weights, which is not zero-shot learning in any standard sense. Third, no code is provided, so the exact preprocessing (grayscale conversion, blur) and training details can't be checked. These are fixable, and they don't refute the claim that a small CNN is competitive; but they do mean the specific margins in Table 7 aren't trustworthy.\n\nCitations look fine—they cite the standard medical imaging and few-shot literature, with no glaring self-citation or citation inflation. The paper is a straightforward empirical report and the authors are transparent about their procedure, which is why I read the main flaw as a protocol error rather than a dishonest one.\n\nWho it's for: anyone working on low-resource medical image classification, or someone teaching what happens when you tune hyperparameters on the test set. It deserves a serious referee—with the understanding that a firmer version needs a validation split, error bars, and a relabeled few-shot section. I'd like to see the numbers re-run before trusting them, but the paper is worth engaging.","headline":"Plausible qualitative result—small CNNs are competitive with pretrained models on these brain-tumor datasets—but the headline accuracies are test-set-selected maxima, so the specific margins need re-running before they can be trusted.","tokens_in":12343,"tokens_out":1940,"would_cite":false,"duration_ms":18747,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-layer CNN rivals pretrained giants on brain-tumor MRI.","keywords":["brain tumor classification","convolutional neural network","ResNet18","VGG16","Br35H dataset","Brain Tumor MRI Dataset","resource-efficient CNN","few-shot learning"],"falsifier":"Re-run the same training protocol with a held-out validation set for the learning-rate sweep, then report test accuracy; and re-measure per-batch inference times on identical hardware. If the custom CNN's accuracy falls notably relative to the pretrained models or the millisecond-level speed advantage disappears, the paper's claims would not survive.","tokens_in":11304,"feed_emoji":"🧠","tokens_out":6309,"duration_ms":49172,"temperature":0.7,"pith_summary":"This paper argues that a deliberately small convolutional network can classify brain tumors from MRI scans nearly as well as large pretrained models while using far fewer parameters and faster inference. On the Br35H binary task the custom CNN reports 98.67% accuracy; on the Brain Tumor MRI Dataset it reports 99.62% for binary and 98.09% for multi-class classification, against ResNet18 and VGG16 scores of roughly 99-100%. The authors conclude that well-designed shallow CNNs are a viable, resource-efficient alternative for medical imaging, and they present few-shot experiments showing the models can still learn from very limited labeled data.","feed_headline":"A two-layer CNN rivals pretrained giants on brain-tumor MRI","feed_subtitle":"Small custom networks score 98.09-99.62% on tumor MRI while running inference in about one millisecond per batch.","key_machinery":"The carrying object is the custom CNN itself: BTBCNN, a two-layer network with 3x3 convolutions (32 then 64 filters), batch normalization, ReLU, max-pooling, a 512-unit dense layer with dropout 0.5, and a single output; and BTMCNN, which adds a third convolutional layer with 128 filters for the four-class task. These models are trained with Adam and binary cross-entropy or cross-entropy loss, with a learning-rate sweep over four values, and compared against ResNet18 and VGG16 under the same preprocessing and 80/20 split. The efficiency argument rests on the reported parameter counts and per-batch inference-time measurements.","core_discovery":"The central claim is that the custom BTBCNN (two convolutional layers) and BTMCNN (three convolutional layers) achieve accuracy competitive with pretrained ResNet18 and VGG16 on the same brain-tumor MRI datasets, while being simpler and computationally cheaper. The paper reports that BTMCNN uses about 51.5 million parameters versus VGG16's 138.4 million, and that per-batch inference times are 0.9-1.4 milliseconds versus 2.8-4.0 milliseconds for the pretrained models. The observed accuracy gap is small (between roughly 0.4 and 1.6 percentage points), which the paper interprets as evidence that model depth and pretraining are not necessary for these classification tasks.","pith_inferences":["A natural next test is whether the same small CNN survives multi-center or cross-scanner MRI data, where domain shift often punishes shallow models more than pretrained deep ones; the paper's single-dataset evaluation cannot answer this.","The few-shot learning curves plateau after roughly 20 shots, so extending the shot counts beyond 80 could reveal whether the remaining gap to full-data accuracy is data-limited or architecture-limited.","If the learning-rate sweep were repeated with a separate validation set, the gap between custom and pretrained models might grow or shrink; this is the most direct way to check whether the headline numbers are robust."],"forward_implications":["A resource-constrained clinic can run brain-tumor MRI screening on a simple CNN and get near-baseline accuracy without pretrained weights or a large GPU cluster.","Under the few-shot protocol, even 80 labeled examples per class push the custom CNNs to roughly 80-90% accuracy on these datasets, suggesting these models are usable in low-data settings.","Because the accuracy gap to ResNet18 and VGG16 is under two percentage points in every comparison, model selection for these specific tasks can reasonably hinge on speed and simplicity rather than accuracy.","The zero-shot results (52%, 69%, 23%) show that without training the models are not useful, so the reported gains are genuinely from learning rather than from dataset bias."],"supporting_citations":[{"why":"Supplies the Br35H binary brain-tumor dataset used for one of the two binary comparisons.","marker":"[8]"},{"why":"Supplies the Brain Tumor MRI Dataset used for both binary and multi-class classification.","marker":"[9]"},{"why":"Defines ResNet18, the pretrained baseline whose accuracy and resource use anchor the comparison.","marker":"[15]"},{"why":"Defines VGG16, the second pretrained baseline and the parameter-count contrast for efficiency claims.","marker":"[33]"},{"why":"Adam, the optimizer used to train every model in the comparison, making the learning-rate sweep meaningful.","marker":"[22]"},{"why":"Cross-entropy loss used for the multi-class BTMCNN training.","marker":"[4]"},{"why":"Binary cross-entropy with logits used for the BTBCNN binary training.","marker":"[31]"}],"fun_headline_variants":["Simple CNN matches deep models on brain tumor MRI","Tiny CNN rivals ResNet and VGG on tumor scans","Two-layer CNN goes head-to-head with big nets on MRI","Lightweight CNN holds its own on brain tumor images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies assume that choosing the learning rate by its score on the final test set does not inflate performance, because no separate validation set was used for model selection.","fun_headline_variants_meta":{"raw":{"variants":["Simple CNN matches deep models on brain tumor MRI","Tiny CNN rivals ResNet and VGG on tumor scans","Two-layer CNN goes head-to-head with big nets on MRI","Lightweight CNN holds its own on brain tumor images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1528,"prompt_tokens":996,"completion_tokens":532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":466}},"tokens_in":612,"tokens_out":532,"duration_ms":5283,"temperature":1.0,"reasoning_tokens":466,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:07:04.715184+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same training protocol with a held-out validation set for the learning-rate sweep, then report test accuracy; and re-measure per-batch inference times on identical hardware. If the custom CNN's accuracy falls notably relative to the pretrained models or the millisecond-level speed advantage disappears, the paper's claims would not survive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Br35H binary brain-tumor dataset used for one of the two binary comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Brain Tumor MRI Dataset used for both binary and multi-class classification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cross-entropy loss used for the multi-class BTMCNN training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Binary cross-entropy with logits used for the BTBCNN binary training."}],"review_version":1}