{"id":"75a152ae-ae15-4521-9cce-703772ad8340","arxiv_id":"2412.20345","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"On a public chest X-ray dataset, VGG19 is reported to outperform SVM, XGBoost, MLP, and ResNet50 in accuracy, AUC, F1, and recall, but without a reproducible evaluation protocol.","lead":"This paper compares five machine learning models, including VGG19, for classifying pneumonia from chest X-ray images and reports that VGG19 gets the highest accuracy and AUC. A generalist might read it as one more benchmark of standard deep learning on a public medical dataset, but the lack of experimental details makes the headline comparison hard to trust.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The VGG19 headline result in Table 1 is numerically identical to the best-optimizer row (AdamW) in Table 2, strongly suggesting that the reported metrics are a selection from an ablation run on the same data rather than a clean held-out test set.","rationale":"The reader correctly identified that the comparison is not demonstrably clean: no train/validation/test split, no repeated runs, no confidence intervals, and no baseline hyperparameters. My stress-test homes in on the stronger, concrete symptom of that same problem: the VGG19 row in Table 1 exactly equals the AdamW row in Table 2. This is not a subtle statistical concern; it indicates that the reported 'VGG19' metrics were likely selected from an optimizer ablation on the same validation data, then presented as the model's independent performance. If the metrics came from a genuine test set, the coincidence of all four values is implausible unless the model and optimizer were identical and the test set was the same as the validation set used for Table 2. The paper also misstates the dataset size (about 50,000 vs 5,863), which is independently verifiable and further undermines the reliability of the experimental description. No machine-checked proofs, code, or data are provided, so the empirical claim cannot be independently checked. Given that the central claim is a benchmark superiority result, and its numeric basis is opaque and likely selection-biased, rejecting the paper is the correct verdict. The manuscript provides no reproducible protocol, no error bounds, and no external validation; the risk of the central claim being false or meaningless is high. I agree with the reader that the evaluation is the weakest point, and my concrete test would settle whether the reported VGG19 result survives a proper held-out evaluation.","tokens_in":7653,"tokens_out":1116,"duration_ms":12499,"concrete_test":"Re-run the comparison with a fixed, publicly documented train/validation/test split (e.g., the standard Kaggle split or a stratified 70/15/15 split with a random seed). For each model, tune hyperparameters only on the validation fold, then report test-set metrics. Verify whether VGG19 still achieves ACC 0.92, AUC 0.95, F1 0.90, recall 0.87 on the test set, and whether those values coincide with the AdamW ablation row. If VGG19's test metrics are lower, or if the gap to baselines shrinks, the headline claim fails.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that VGG19 outperforms four baselines on all metrics for pneumonia X-ray classification (Section IV-B, Table 1). The single most load-bearing fragility is that these numbers are unverifiable because no data-splitting protocol is described: the paper never states how images were partitioned into training, validation, and test sets, nor whether Table 1 reports performance on an independent test set. The paper is therefore internally consistent only if one assumes a fair test split. That assumption is directly undermined by the paper's own text and tables: Table 2 (optimizer ablation) reports AdamW with ACC 0.92, AUC 0.95, F1 0.90, recall 0.87, exactly the same four values as the VGG19 row in Table 1. This suggests the 'final' VGG19 result was chosen as the best-performing optimizer configuration from an ablation on validation data and then reported as the model's headline performance. Such selection-on-the-same-data invalidates the comparison: the baselines (SVM, XGBoost, MLP, ResNet50) are not reported with the same optimizer tuning or with any hyperparameter details, so the 2-point accuracy gaps may reflect tuning advantage rather than architectural superiority. Additionally, the dataset is stated to contain about 50,000 images (Section IV-A), while the public Kaggle 'Chest X-ray Images (Pneumonia)' dataset has 5,863 images; this factual error further weakens confidence in the experimental description. The conclusion that VGG19 is best therefore rests on an unverifiable and likely selection-based evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a comparison of five classifiers for pneumonia detection in chest X-ray images: SVM, XGBoost, MLP, ResNet50, and VGG19. Using what it describes as the public Kaggle 'Chest X-ray Images (Pneumonia)' dataset, it claims in the abstract and in Section IV-B that VGG19 achieves the best performance on all four reported metrics: accuracy 0.92, AUC 0.95, F1 score 0.90, and recall 0.87. The manuscript provides a standard description of VGG19's convolution, pooling, fully connected, and softmax layers, followed by an experimental section containing a results table, an optimizer ablation table, and a loss-curve figure. The paper's central conclusion is that deep convolutional networks, particularly VGG19, are superior to the compared baselines on this medical imaging task.","tokens_in":7892,"tokens_out":2745,"duration_ms":29471,"significance":"If the reported comparison were trustworthy, the result would be a modest, incremental confirmation of a widely held expectation: deep CNNs outperform classical machine-learning pipelines on chest X-ray classification. The paper's contribution is therefore not conceptual novelty but an empirical benchmark. However, the manuscript provides no code, no trained models, no data-splitting protocol, no repeated runs, and no confidence intervals, so the benchmark cannot be independently reproduced or statistically assessed. The numerical coincidence between the VGG19 row in Table 1 and the AdamW row in Table 2, combined with the incorrect dataset size in Section IV-A, casts direct doubt on the validity of the headline result. The paper's strength is limited to a clear description of the VGG19 architecture; it does not ship machine-checked proofs, reproducible code, or falsifiable predictions beyond the reported point estimates.","major_comments":[{"comment":"The load-bearing claim that VGG19 outperforms all baselines rests on metrics that are not shown to come from an independent test set. The paper never states how the data were partitioned into training, validation, and test sets, nor whether Table 1 reports performance on a held-out test split. More specifically, the VGG19 row in Table 1 (ACC 0.92, AUC 0.95, F1 0.90, recall 0.87) is numerically identical to the AdamW row in Table 2, yet the text does not explain that AdamW was selected from the ablation and then evaluated on a separate test set. The reported numbers are therefore consistent with selecting the best optimizer on validation data and presenting that same selection as the model's headline performance, which invalidates the comparison because the baselines are not given equivalent tuning.","section":"Section IV-B, Tables 1 and 2"},{"comment":"The dataset description is factually incorrect: the text states that the public Kaggle 'Chest X-ray Images (Pneumonia)' dataset contains about 50,000 images, but the commonly referenced version of this dataset contains 5,863 images. This is not a minor typo; it changes the scale of the experiment and prevents readers from reconstructing the training conditions. The authors should state the exact number of images, the number of patients, the class balance, and the official split (if the standard train/val/test split was used) or report their own split explicitly.","section":"Section IV-A"},{"comment":"The experimental protocol is too underspecified to support the central comparison. No random seed, no data-augmentation details, no preprocessing parameters (beyond 'scaled, cropped, and normalized'), no optimizer settings for the baselines, no learning-rate schedule for models other than VGG19, and no repeated runs or confidence intervals are reported. Without this information, the two-to-five percentage point gaps between VGG19 and the baselines in Table 1 cannot be distinguished from tuning artifacts or noise. The authors should report the exact protocol for every model, including hyperparameter search ranges and the selected configurations.","section":"Section IV"}],"minor_comments":[{"comment":"The convolution formula uses the symbol 'E' to denote convolution, but the surrounding text and standard notation imply that '*' or '\\ast' is intended; this should be corrected for readability.","section":"Section III, equations"},{"comment":"The pooling equation reuses the notation I_l_pool for both the input feature map and the output, making the formula ambiguous. Please use distinct symbols for the input and output of the pooling operation.","section":"Section III, pooling formula"},{"comment":"Figure 2 is described as showing training and validation loss versus epoch, but the axis labels and legend are not legible in the manuscript; please provide a high-resolution figure with clearly labeled axes and a caption that states which optimizer's curves are shown.","section":"Figure 2"},{"comment":"Several references are to arXiv preprints and appear only loosely related to the specific claims in the text; for claims about pneumonia classification, please cite peer-reviewed sources where available.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's central claim is not supported by the evidence as reported. The numerical identity between Table 1's VGG19 row and Table 2's AdamW row, together with the absence of any stated data-splitting protocol, means the main comparison is unverifiable. The factual error on dataset size further erodes confidence. A major revision could in principle address these issues by rerunning or fully documenting the experiment, but the current manuscript would require new experimental evidence rather than textual fixes, which in my view places it outside the scope of a revision for this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a straightforward benchmark: VGG19, ResNet50, MLP, XGBoost, and SVM on the public Kaggle chest X-ray pneumonia dataset, with four metrics reported and an optimizer ablation. That structure is fine for a technical report, and the method section describes VGG19 competently. The writing is clear, and the citations to prior VGG19 pneumonia work are relevant. Credit is due for including several baselines and for reporting AUC, F1, and recall, not just accuracy.\n\nThe problem is that the load-bearing result is almost certainly an artifact of selection on the validation set. The VGG19 row in Table 1 (0.92 / 0.95 / 0.90 / 0.87) is numerically identical to the AdamW row in Table 2. That is not a coincidence; it means the \"final\" test metrics are the best-performing configuration from the optimizer ablation, reported without any indication that they came from a clean held-out test split. The paper never states how the data were partitioned, gives no repeated runs, no confidence intervals, and no baseline hyperparameter details. The baselines were not given the same tuning opportunity, so the 2–3 point gaps between VGG19 and ResNet50 could easily reflect tuning effort rather than architectural merit.\n\nThe dataset-size error is also real: the text claims about 50,000 images, but the public Kaggle dataset has 5,863. That is a factual mistake in the experimental description and further reduces confidence in the numbers. The novelty is minimal, since references [2] and [3] already apply VGG19 to the same task.\n\nTo be fair, the broad claim that VGG19 can classify pneumonia X-rays well is consistent with prior work and probably true. But the specific claim that VGG19 beats the other models on all metrics is not supported by the evidence as presented. The identical table values and missing protocol make the evaluation unverifiable, and I would not trust the comparison without a completely rewritten experimental section that includes a proper train/val/test split, repeated runs, and honest reporting of whether the reported numbers came from validation or test.\n\nThis is not worth a serious referee in its current form. It is the kind of paper that should be desk-rejected and sent back with a clear request for a rigorous evaluation, or allowed to die quietly. I would not cite it, and I would not bring it to reading group.","headline":"Routine VGG19 benchmark on pneumonia X-rays with an evaluation flaw that sinks the comparative claim: the headline numbers are identical to the best-validation optimizer row.","tokens_in":8530,"tokens_out":2443,"would_cite":false,"duration_ms":24900,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that VGG19 beats SVM, XGBoost, MLP, and ResNet50 on pneumonia X-ray classification, with 92% accuracy and 0.95 AUC.","keywords":["VGG19","pneumonia classification","chest X-ray","convolutional neural networks","deep learning","transfer learning","model comparison","medical image classification"],"falsifier":"Retrain all five models on the public chest X-ray dataset (5,863 images, not the roughly 50,000 the paper describes) with a documented train/validation/test split, identical preprocessing, and identical optimizer settings; if VGG19's 92% accuracy and 0.95 AUC do not beat ResNet50 on all four metrics, the central claim is refuted.","tokens_in":7378,"feed_emoji":"🩻","tokens_out":5547,"duration_ms":52167,"temperature":0.7,"pith_summary":"The paper sets out to establish that a standard deep convolutional network, VGG19, can classify pneumonia from chest X-rays more reliably than four alternative machine-learning and deep-learning models. On its reported results, VGG19 reaches 92% accuracy, 0.95 AUC, 0.90 F1, and 0.87 recall, beating support vector machines, XGBoost, a multilayer perceptron, and ResNet50 on every measured metric. The practical stake is that a simple, well-understood architecture with pre-trained weights could support automated pneumonia screening, provided the evaluation is as clean as the paper assumes.","feed_headline":"VGG19 beats four rivals on pneumonia X-ray classification","feed_subtitle":"Reported 92% accuracy and 0.95 AUC put a standard CNN ahead of SVM, XGBoost, MLP, and ResNet50.","key_machinery":"The machinery is VGG19, a 19-weight-layer convolutional neural network built from stacked convolutional and max-pooling layers followed by fully connected layers and a softmax output. The argument runs through this architecture's feature extraction: each convolution layer pulls edges, textures, and higher-level shapes from the X-ray, pooling reduces dimensionality, and the final layers map the feature vector to a pneumonia-or-normal decision, trained with cross-entropy loss and backpropagation.","core_discovery":"The paper's central claim is that VGG19, a 19-weight-layer convolutional neural network, outperformed all four comparison models on the public Chest X-ray Images (Pneumonia) dataset, achieving 92% accuracy, 0.95 AUC, 0.90 F1, and 0.87 recall. The authors report that ResNet50 came closest but fell behind on recall and F1, while SVM, XGBoost, and MLP were weaker across the board. The paper also presents an optimizer ablation in which AdamW produced exactly the same top metrics as the main VGG19 result, leading the authors to attribute the outcome to the combination of VGG19's feature extraction and AdamW's adaptive learning rate with weight decay.","pith_inferences":["Because the paper never states the train/validation/test split, the number of runs, or the baseline hyperparameters, the reported margin is best read as a demonstration of VGG19's potential rather than a proven ordering; a controlled re-run is the natural next check.","The paper describes the dataset as containing about 50,000 images, while the public release has roughly 5,863; that discrepancy matters because the reported metrics cannot be reproduced without knowing exactly which images and which split were used.","If the result holds after a cleaner evaluation, the practical takeaway is that disciplined evaluation may matter more than model choice: a standard CNN fine-tuned with AdamW could be enough for a deployable screening tool."],"forward_implications":["If the reported ordering is correct, pneumonia screening from chest X-rays can be automated with an off-the-shelf deep network rather than hand-engineered features.","The results imply that conventional machine-learning classifiers, applied directly to images, are not competitive with deep convolutional networks on this task.","The optimizer ablation suggests that AdamW, not just the network architecture, is responsible for the top scores, since the best VGG19 row matches the AdamW row exactly.","ResNet50's lower recall and F1 would mean it misses more borderline pneumonia cases than VGG19 would under the paper's measurements."],"supporting_citations":[{"why":"It supplies the VGG19 architecture and its prior application to pneumonia prediction, which this study extends.","marker":"[2]"},{"why":"It supplies the enhanced convolutional neural network approach to pneumonia detection that frames the training methodology.","marker":"[24]"},{"why":"It supplies the support vector machine baseline whose pneumonia classification results are compared against VGG19.","marker":"[25]"},{"why":"It supplies the XGBoost baseline whose pneumonia detection results are compared against VGG19.","marker":"[26]"},{"why":"It supplies the multilayer perceptron baseline whose pneumonia diagnosis results are compared against VGG19.","marker":"[27]"},{"why":"It supplies the ResNet50 baseline whose pneumonia classification results are compared against VGG19.","marker":"[28]"}],"fun_headline_variants":["VGG19 outperforms four rivals on pneumonia X-rays","VGG19 hits 92% accuracy on pneumonia X-ray images","Pneumonia X-ray model: VGG19 beats all four baselines","VGG19 leads pneumonia X-ray classification with 0.95 AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison collapses if the reported metrics did not come from a clean, held-out test set; the paper never states how the data were split, how many runs were averaged, or what hyperparameters the baseline models used.","fun_headline_variants_meta":{"raw":{"variants":["VGG19 outperforms four rivals on pneumonia X-rays","VGG19 hits 92% accuracy on pneumonia X-ray images","Pneumonia X-ray model: VGG19 beats all four baselines","VGG19 leads pneumonia X-ray classification with 0.95 AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1569,"prompt_tokens":927,"completion_tokens":642,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":566}},"tokens_in":543,"tokens_out":642,"duration_ms":6229,"temperature":1.0,"reasoning_tokens":566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:23:46.213587+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain all five models on the public chest X-ray dataset (5,863 images, not the roughly 50,000 the paper describes) with a documented train/validation/test split, identical preprocessing, and identical optimizer settings; if VGG19's 92% accuracy and 0.95 AUC do not beat ResNet50 on all four metrics, the central claim is refuted.","supporting_citations":[{"cited_title":"A deep learning model for early prediction of pneumonia using VGG19 and neural networks,","cited_arxiv_id":null,"evidence_quote":"It supplies the VGG19 architecture and its prior application to pneumonia prediction, which this study extends."},{"cited_title":"Pneumonia detection using enhanced convolutional neural network model on chest X -ray images,","cited_arxiv_id":null,"evidence_quote":"It supplies the enhanced convolutional neural network approach to pneumonia detection that frames the training methodology."},{"cited_title":"Classification of pneumonia from chest X -ray images using support vector machine and convolutional neural network,","cited_arxiv_id":null,"evidence_quote":"It supplies the support vector machine baseline whose pneumonia classification results are compared against VGG19."},{"cited_title":"Pneumonia detection in chest X -ray images using an optimized ensemble with XGBoost classifier,","cited_arxiv_id":null,"evidence_quote":"It supplies the XGBoost baseline whose pneumonia detection results are compared against VGG19."},{"cited_title":"Enhanced pneumonia diagnosis using chest X- ray image features and multilayer perceptron and k -NN machine learning algorithms,","cited_arxiv_id":null,"evidence_quote":"It supplies the multilayer perceptron baseline whose pneumonia diagnosis results are compared against VGG19."},{"cited_title":"Classification of pneumonia based on X -ray images with ResNet -50 architecture,","cited_arxiv_id":null,"evidence_quote":"It supplies the ResNet50 baseline whose pneumonia classification results are compared against VGG19."}],"review_version":1}