{"id":"c8fa26f2-1690-499e-a47c-7a5a34e2c215","arxiv_id":"2501.14279","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"Transfer learning with AlexNet, ResNet152, and InceptionV3 on a small chest X-ray subset improves AUC over a random baseline, but the reported F1 gains are contradicted by the paper's own tables.","lead":"This paper reports that fine-tuning ImageNet-pretrained neural networks on a 10% subset of the NIH ChestX-ray14 dataset raises AUC for thoracic disease classification. It also claims a 15% F1-score gain, but its own tables show F1 falling for every model.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's claimed 15% F1 improvement for InceptionV3 is contradicted by the paper's own Tables 1 and 2, where F1 drops from 0.287 to 0.156.","rationale":"The reader's strongest claim correctly identifies the abstract's F1 assertion as unsupported. My stress-test goes further: the F1 assertion is not merely unsupported but is contradicted by the paper's own Tables 1 and 2. The reader's weakest assumption was that the 10% mini-dataset is a faithful proxy for the full NIH ChestX-ray14 dataset; that is a legitimate external-validity concern, but the decisive problem is internal inconsistency, not representativeness. Even if the mini-dataset were perfectly representative, the headline F1 improvement would still be false on the reported numbers. The AUC improvement from 0.500 to 0.639 deserves credit as a real, if modest, gain over random baseline. However, the abstract presents both improvements as equal headline achievements, and the F1 half is demonstrably wrong. In addition, the paper attributes gains to focal loss while the training configuration in Section 4.3 says BCEWithLogitsLoss was used, with no focal loss parameters anywhere; this weakens the mechanistic explanation. Since the central claim of the paper is exactly this transfer-learning-plus-focal-loss improvement, and the paper's own evidence contradicts a key part of it, the rejection verdict stands unchanged. No further external benchmark or re-analysis is needed to establish that the abstract is not supported by the results as reported.","tokens_in":6781,"tokens_out":2539,"duration_ms":22061,"concrete_test":"Recompute the F1 deltas from the published tables. For InceptionV3, if Table 1 F1=0.287 and Table 2 F1=0.156 are correct, then ΔF1=−0.131 (−45.6%). If the claimed +15% increase is correct, Table 2 should read F1≈0.330. The authors should provide the per-class confusion matrices or raw prediction logs for the InceptionV3 test split and recompute F1=2TP/(2TP+FP+FN); whichever result emerges, the abstract or the tables must be corrected. Also verify whether focal loss was actually used during training, since Section 4.3 states BCEWithLogitsLoss was the loss function.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract—'The InceptionV3 model ... achieved a 28% improvement in AUC and a 15% increase in F1-Score'—is internally contradicted by the paper's own experimental tables. Baseline Table 1 (Section 4.2) reports InceptionV3 F1 = 0.287, AUC = 0.500. Post-fine-tuning Table 2 (Section 4.4) reports InceptionNet F1 = 0.156, AUC = 0.639. The AUC gain is real: 0.639/0.500 − 1 ≈ 27.8%, consistent with a 28% improvement. But the F1 change is 0.156 − 0.287 = −0.131, a relative decrease of roughly 45.6%, not a 15% increase. Because the abstract presents the F1 gain as a headline result and the only table reporting F1 contradicts it, the paper's central quantitative claim fails on its own evidence. This is not a matter of dataset representativeness or external validity; it is an internal numerical inconsistency that no external benchmark can repair unless one of the tables or the abstract is corrected. Relatedly, the text credits focal loss (Sections 3.2 and 4.4.1) for handling class imbalance, but the actual training configuration in Section 4.3 lists BCEWithLogitsLoss as the loss function and gives no focal-loss hyperparameters. This further undermines the causal attribution in the abstract. Both issues make the 'significant performance improvements' claim unsubstantiated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies transfer learning with AlexNet, ResNet152, and InceptionV3 to a 10% subset of the NIH ChestX-ray14 dataset for multi-label thoracic disease classification. It reports baseline (Table 1) and post-fine-tuning (Table 2) values of BCE loss, F1-score, and AUC, and uses Grad-CAM to visualize model attention. The central claim is that fine-tuning with focal loss yields significant improvements, with InceptionV3 achieving a 28% AUC improvement and a 15% F1-score increase. The paper concludes that the approach is promising but not yet state-of-the-art.","tokens_in":7052,"tokens_out":2946,"duration_ms":27088,"significance":"If the reported improvements were reliable, the paper would provide a straightforward demonstration that standard transfer learning helps on a small chest X-ray sample, with interpretability as a secondary contribution. The use of a public dataset and three standard architectures is sensible, and the Grad-CAM figures illustrate an interpretability pipeline that could be useful for teaching or prototyping. However, the experimental evidence as presented is not trustworthy: the headline F1 claim is directly contradicted by the paper's own tables, the loss-function attribution is inconsistent with the training configuration, and there is no validation protocol, error analysis, or comparison to established benchmarks. Because the central quantitative claim fails on internal evidence and the remaining results are modest point estimates without statistical support, the paper does not currently meet the bar for a serious journal publication.","major_comments":[{"comment":"The abstract's claim that InceptionV3 achieved a 15% increase in F1-Score is contradicted by the paper's own tables. Table 1 reports InceptionV3 baseline F1 = 0.287, while Table 2 reports post-fine-tuning InceptionNet F1 = 0.156. This is a relative decrease of approximately 45.6%, not a 15% increase. The AUC claim (0.500 to 0.639, about 27.8% relative improvement) is consistent with the tables, but the F1 claim is a load-bearing part of the abstract and is false on the manuscript's own evidence. The paper must either correct this claim or provide the missing experimental record that supports it.","section":"Abstract and §4.2/§4.4, Tables 1 and 2"},{"comment":"The paper attributes performance improvements to focal loss for class imbalance, but the training configuration in §4.3 lists BCEWithLogitsLoss as the loss function and gives no focal-loss hyperparameters (alpha, gamma). Table 2 reports a column labeled 'F Loss', yet the text never specifies how focal loss was implemented, what hyperparameters were used, or how it interacts with the BCE loss. The causal statement in §4.4.1 that improvements highlight 'the importance of Focal Loss' is therefore unsupported by the described experimental setup.","section":"§3.2, §4.3, and §4.4.1"},{"comment":"No validation protocol is described. The paper states hyperparameters such as learning rate schedule, batch size, number of epochs, and fine-tuning layer cuts, but it does not state how these were selected or whether a held-out validation set was used. Without such a protocol, the reported test-set numbers may reflect hyperparameter selection on the test set, and the absence of error bars or confidence intervals means that the differences between models in Table 2 (e.g., F1 0.1401 vs 0.1596) cannot be assessed statistically. This is a load-bearing omission for any claim of 'significant performance improvements.'","section":"§4.3 and §4.4"},{"comment":"The 10% mini-dataset is described only as 'curated' while preserving the original 80/20 split, but the paper gives no evidence that class frequencies, patient-level separation, or image-quality distributions are preserved. NIH ChestX-ray14 contains multiple images per patient, and if the mini-dataset split does not respect patient boundaries, the reported AUC and F1 values can be optimistically biased by patient leakage. The representativeness assumption is load-bearing for transferring the conclusions to the full dataset, and it is not demonstrated.","section":"§2.2 and §4.2"},{"comment":"Even taken at face value, the post-fine-tuning results are weak: the best AUC in Table 2 is 0.6397 (AlexNet), barely above random for a multi-label chest X-ray task, and the F1 values decrease relative to baseline for every model. The conclusion acknowledges the models are not state-of-the-art, but the paper does not compare against any existing benchmark or per-class results, making it impossible to judge whether the reported values are meaningful for the 14 disease classes. The central narrative of 'significant performance improvements' is not supported by the reported metrics.","section":"§4.4 and §6"}],"minor_comments":[{"comment":"The sentence 'Acquired from Kaggle's NIH ChestX-ray14 dataset' is repeated almost verbatim later in the same paragraph; one occurrence should be deleted.","section":"§2.1"},{"comment":"The description of data augmentation (random cropping and horizontal flipping) is repeated several times in the same subsection; the duplication should be removed.","section":"§2.3"},{"comment":"There is a typo in 'This indicatesthe models were nearing their optimal performance'; a space is missing.","section":"§4.4.2"},{"comment":"The caption contains 'middle figuree' and incorrectly labels the third image as both 'Final Convolution Layer' and 'middle figure'; the caption should be corrected and made consistent with the figure panels.","section":"Figure 4 caption"},{"comment":"The claim that ResNet showed a 20% reduction in BCE loss is not consistent with Table 2: ResNet152 BCE loss decreases from 0.928 to 0.7945, which is about 14.4%, not 20%. The sentence should either be corrected or the relevant numbers should be reported explicitly.","section":"§4.4"},{"comment":"Reference [14] is incomplete (missing authors' full names and correct title) and reference [16] appears to include an implausible author list; the bibliography should be checked against the original sources.","section":"References"}],"recommendation":"reject","confidential_remarks":"The internal contradiction between the abstract's F1 claim and Tables 1 and 2 is decisive, and the additional protocol problems (no validation set, no error bars, loss-function mismatch) would require a substantially new experimental design rather than a simple revision. The manuscript reads more like a course project report than a journal article; in addition, the data and code availability statement ('available from the corresponding author upon reasonable request') is not verifiable and does not meet reproducibility expectations for empirical work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a routine transfer-learning benchmark on a 10% slice of NIH ChestX-ray14, and the abstract's headline F1 claim is contradicted by the paper's own tables. I'd let it go.\n\nWhat's actually there: three standard architectures (AlexNet, ResNet152, InceptionV3), ImageNet initialization, 20 epochs of fine-tuning, BCEWithLogitsLoss, and Grad-CAM. The one real result is that fine-tuning lifts AUC from around 0.5 to 0.64 for all models. That is consistent with a well-known effect and is not new. The Grad-CAM figures show the models attend to plausible lung regions, which is nice but not a contribution.\n\nThe soft spots are load-bearing. The abstract says InceptionV3 achieved a 15% increase in F1-Score; Table 1 gives baseline F1 0.287 and Table 2 gives post-fine-tune F1 0.156. That is a 45% decrease, not a 15% increase. The AUC number (0.500 to 0.639) matches a 28% relative gain, so the abstract mixed a real AUC gain with a fabricated F1 claim. The text also credits focal loss for handling class imbalance, but Section 4.3 says the loss was BCEWithLogitsLoss and no focal-loss hyperparameters appear anywhere. There are no error bars, no held-out validation for hyperparameters, no patient-level split, and no code or data release. The 10% subset is described as 'curated' but there is no evidence the class frequencies or patient splits are preserved.\n\nNone of this is fixable by a half-hearted revision except the internal numbers, but the surrounding issues — no validation protocol, no code, no new method — mean the paper would not teach a reader anything not already in the literature. It reads like a course-project writeup, not a research contribution, though the authors do honestly admit in the conclusion that the models are below state-of-the-art.\n\nRecommendation: desk reject. If the authors fix the abstract/table mismatch and add a proper validation split with error bars and code, it might be a reasonable workshop poster, but it does not deserve referee time at a serious venue.","headline":"Routine transfer-learning benchmark on a 10% NIH ChestX-ray14 subset, and the abstract's headline F1 claim is contradicted by the paper's own tables; not worth refereeing.","tokens_in":7659,"tokens_out":2469,"would_cite":false,"duration_ms":20651,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper sets out to show that fine-tuning ImageNet-pretrained networks with focal loss materially improves multi-label thoracic disease classification on chest X-rays, reporting InceptionV3's AUC rising from 0.500 to 0.6390 on a 10% NIH…","keywords":["Chest X-ray classification","Transfer learning","Focal loss","Grad-CAM","Class imbalance","Multi-label classification","NIH ChestX-ray14","Thoracic diseases"],"falsifier":"A reader could compute the class-frequency and patient-overlap statistics of the 10% subset against the full NIH ChestX-ray14 dataset, then rerun the same fine-tuning on the full training split and evaluate on the full test split; if the AUC gain shrinks toward baseline or the F1 decline persists, the mini-dataset results do not generalize. A simpler check: identify the exact patient identifiers in the mini train and test sets; any patient appearing in both would inflate every reported metric.","tokens_in":6502,"feed_emoji":"🩻","tokens_out":7016,"duration_ms":57639,"temperature":0.7,"pith_summary":"This paper tries to establish that a standard transfer-learning recipe—ImageNet pre-trained convolutional networks fine-tuned on a small chest X-ray sample with focal loss—can improve automated detection of thoracic diseases, and that Grad-CAM heatmaps show the models look at clinically relevant lung regions. The authors evaluate AlexNet, ResNet152, and InceptionV3 on a 10% curated subset of the NIH ChestX-ray14 dataset and report that fine-tuning raised every model's AUC, with InceptionV3 rising from 0.500 to 0.6390. The abstract also claims a 15% increase in InceptionV3's F1-score, although Tables 1 and 2 show F1 falling from 0.287 to 0.1560. If the AUC gains are real, they would suggest that relatively cheap transfer learning can produce useful disease-ranking behavior even with severe class imbalance and limited data. The authors frame the contribution as feasibility evidence, not state-of-the-art performance.","feed_headline":"Fine-tuning lifts chest X-ray AUC to 0.64 from 0.50","feed_subtitle":"ImageNet-pretrained CNNs with focal loss improve disease ranking on a small NIH ChestX-ray14 subset, though F1 gains don't follow.","key_machinery":"The mechanism is transfer learning on ImageNet-pretrained backbones—AlexNet, ResNet152, and InceptionV3—combined with focal loss and Grad-CAM visualization. Focal loss is a modified cross-entropy that down-weights well-classified examples, redirecting gradient toward the underrepresented disease classes that dominate chest X-ray datasets. Transfer learning supplies low-level feature detectors learned from ImageNet, which the authors argue helps with small medical datasets, while Grad-CAM generates class-specific heatmaps from the final convolutional layer to show which image regions drive each prediction. The claimed result depends on these three components working together: pretrained features, imbalance-correcting loss, and localization-based interpretability.","core_discovery":"The central claim is that fine-tuning ImageNet-pretrained architectures with focal loss yields significant performance gains for multi-label chest X-ray classification on the NIH ChestX-ray14 dataset. On the 10% mini-dataset, InceptionV3's AUC improved from 0.500 at baseline to 0.639 after fine-tuning, a 28% relative improvement, and AlexNet's AUC rose from 0.469 to 0.6397. The abstract additionally reports a 15% F1-score increase for InceptionV3, but the tables list F1 falling from 0.287 to 0.1560, so that particular claim is not supported by the reported numbers. The paper presents Grad-CAM visualizations for positive atelectasis and effusion as evidence that the models attend to diagnostically relevant areas, and explicitly concludes that the models remain short of state-of-the-art performance.","pith_inferences":["The 28% AUC gain is a relative improvement from a baseline of 0.500, which is random guessing; starting from a stronger baseline would likely show smaller relative gains, so the headline number should be read in context.","The F1-Score decline between Table 1 and Table 2 is not consistent with the abstract's '15% increase' claim; treating that claim as unverified until a matching evaluation appears would be prudent.","Because the 10% subset's class frequencies and patient-level separation are not characterized, the reported metrics could shift materially on the full dataset; a direct full-dataset evaluation is the natural next test.","Grad-CAM images are suggestive evidence of clinical alignment, but without quantitative comparison to radiologist annotations or segmentation masks, the interpretability claim remains to be demonstrated."],"forward_implications":["If the reported AUC gains are correct, transfer learning with focal loss can take small chest X-ray datasets from near-random ranking to moderately useful screening performance without training from scratch.","Grad-CAM alignments with atelectasis and effusion regions, if consistent across cases, would indicate the fine-tuned models are using anatomical evidence rather than dataset artifacts.","Because the models improved AUC more than F1, their practical value would lie in ranking and triage rather than in balanced precision/recall diagnosis.","The mini-dataset protocol could serve as a fast experimental testbed for the full NIH ChestX-ray14 dataset, since the same label mapping and train/test split are retained.","The authors' own conclusion implies the immediate contribution is feasibility evidence, not a clinically deployable system."],"supporting_citations":[{"why":"Supplies the NIH ChestX-ray14 images and multi-label annotations that all experiments use.","marker":"[11]"},{"why":"Focal loss, the loss function the paper credits for addressing class imbalance.","marker":"[4]"},{"why":"AlexNet, one of the three transfer-learned architectures evaluated.","marker":"[3]"},{"why":"ResNet152, the deep skip-connection architecture used for feature extraction and fine-tuning.","marker":"[1]"},{"why":"InceptionV3, the multi-scale architecture highlighted in the abstract's headline AUC claim.","marker":"[14]"},{"why":"Grad-CAM, the interpretability method used to produce the localization heatmaps.","marker":"[9]"}],"fun_headline_variants":["Fine-tuned InceptionV3 lifts chest X-ray AUC by 28% on mini-set","Chest X-ray ranking improves with focal loss, but F1 drops","AUC up 28%, F1 down: CNN fine-tuning on X-rays yields mixed results","Small X-ray set: deep learning AUC gains don't translate to F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 10% mini-dataset faithfully represents the full NIH ChestX-ray14 dataset, preserving class frequencies, image-quality distribution, and the original 80/20 patient split, so metrics measured on this subset transfer to the full dataset.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned InceptionV3 lifts chest X-ray AUC by 28% on mini-set","Chest X-ray ranking improves with focal loss, but F1 drops","AUC up 28%, F1 down: CNN fine-tuning on X-rays yields mixed results","Small X-ray set: deep learning AUC gains don't translate to F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000728,"raw_usage":{"total_tokens":3231,"prompt_tokens":887,"completion_tokens":2344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2254}},"tokens_in":503,"tokens_out":2344,"duration_ms":16814,"temperature":1.0,"reasoning_tokens":2254,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:14:48.576743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could compute the class-frequency and patient-overlap statistics of the 10% subset against the full NIH ChestX-ray14 dataset, then rerun the same fine-tuning on the full training split and evaluate on the full test split; if the AUC gain shrinks toward baseline or the F1 decline persists, the mini-dataset results do not generalize. A simpler check: identify the exact patient identifiers in the mini train and test sets; any patient appearing in both would inflate every reported metric.","supporting_citations":[{"cited_title":"Roth, Zhiyun Gao, Liyue Lu, Ziyue Xu, Isaac Nogues, Jianhua Yao, Daniel Mollura, and Ronald M","cited_arxiv_id":null,"evidence_quote":"Supplies the NIH ChestX-ray14 images and multi-label annotations that all experiments use."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AlexNet, one of the three transfer-learned architectures evaluated."},{"cited_title":"Rethinking the inception architecture for computer vision","cited_arxiv_id":null,"evidence_quote":"InceptionV3, the multi-scale architecture highlighted in the abstract's headline AUC claim."},{"cited_title":"Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra","cited_arxiv_id":null,"evidence_quote":"Grad-CAM, the interpretability method used to produce the localization heatmaps."}],"review_version":1}