{"id":"fdbd8728-ba2b-4857-aaab-577ea0b1a817","arxiv_id":"2411.11376","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A Vision Transformer on chest X-rays is claimed to reach 97.83% accuracy, but the reported validation accuracy and missing baselines do not support this claim.","lead":"This preprint fine-tunes a Vision Transformer on chest X-rays to classify lung diseases, comparing full images with lung-segmented images. The headline accuracy claims conflict with the paper's own result tables, so the main finding is not established.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline accuracy figures (97.83%, 96.58%) do not appear anywhere in the reported results; the only stated validation accuracy is 74.52% for the region-focused model, so the central claim is unsupported by the manuscript's own experiments.","rationale":"The reader's REJECT verdict is sound. The most load-bearing issue is simpler and more fundamental than the patient-split concern: the numbers that define the paper's contribution are not present in the results section. A comparative study that claims 97.83% accuracy must show that number in a table or figure with the corresponding split; no such value appears. The patient-split issue would additionally invalidate the comparison if images from the same patient straddle train/test, but even with a perfect split, the paper fails to report the accuracy used for its headline. Therefore the central claim is unverifiable as written. The agreement is partial because the reader focused on data leakage, while we identify the missing quantitative support; both point to REJECT, but the missing results are the more direct falsifier. The concrete test above would settle it by attempting to reproduce the 97.83% figure.","tokens_in":845,"tokens_out":849,"duration_ms":35266,"concrete_test":"Conduct an independent re-derivation of the reported results: fine-tune google/vit-base-patch16-224-in21k on the described NIH/COVID-19 subset (12,897 training and 2,975 test images, seven classes) with a patient-level split, 10 epochs, batch size 32, AdamW lr=1e-4, and compute test-set accuracy, precision, recall, F1, and AUC. If the test accuracy does not reach 97.83% (or if the 97.83% figure cannot be located in the authors' own logs), the abstract's central claim is not reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, that a full-image ViT reaches 97.83% accuracy and a lung-segmented ViT reaches 96.58%, is not evidenced by the experimental section. Section V (Results) reports only per-epoch loss, ROC AUC, and MCC for the two ViT variants (Tables III and IV); no accuracy, precision, recall, or F1 values are tabulated, and Figure 5 ('Accuracy, Precision, Recall, and F1 Scores') is neither described nor sourced. The only numerical accuracy stated is 'a validation accuracy of 74.52%' in Section V-A, and the surrounding text attributes it to Table IV even though the value matches Table III's final row, so the experimental reporting is internally inconsistent. Even accepting the AUC/MCC numbers, the abstract's 97.83% and 96.58% accuracy claims are absent from the results, and no CNN baseline is trained under the same protocol. The abstract also shifts among 'three label,' 'seven categories,' and 'eight' labels, and the model description reports 597,511 total parameters for a ViT-base with a 590,592-parameter patch embedding and 5,383-parameter head (the sum is 595,975, not 597,511), further indicating that the quantitative statements are not reliable. The load-bearing assumption for the claimed superiority is therefore not just the train/test split; it is that the reported accuracy numbers exist at all, and they do not.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims to compare two Vision Transformer (ViT) approaches, one trained on full chest X-ray images and one on lung-segmented images, and states in the abstract and conclusion that the full-image ViT reaches 97.83% accuracy, the lung-segmented ViT reaches 96.58% accuracy, and that an AUC of 94.54% is achieved when label counts are increased to eight. The experimental section, however, reports only per-epoch loss, ROC AUC, and MCC for both models, with no accuracy, precision, recall, or F1 values in the tables. The single numerical accuracy stated in the text, 74.52%, is attributed to Table IV but appears to match Table III. The paper does not include any CNN baseline trained under the same protocol, and the dataset split is not described at patient level. The conclusion repeats the unsupported accuracy figures.","tokens_in":6984,"tokens_out":2311,"duration_ms":22558,"significance":"If the headline accuracy figures were supported by reproducible experiments, the claim that a standard ViT fine-tuned on full chest X-rays outperforms both CNN baselines and a lung-segmentation pipeline would be of moderate interest to the medical imaging community. The manuscript, however, does not substantiate these claims with its own reported results, does not provide error bars or statistical significance, and omits a CNN comparison. The central quantitative claims therefore currently rest on no evidentiary basis within the paper. The study also uses the public NIH ChestX-ray8 and COVID-19 datasets, which are widely studied, but the experimental reporting is too incomplete to allow replication or fair comparison.","major_comments":[{"comment":"The abstract and conclusion claim that the full-image ViT achieves 97.83% accuracy and the lung-segmented ViT achieves 96.58% accuracy, but these numbers appear nowhere in Section V. Tables III and IV report only Loss, ROC AUC, and MCC per epoch; no accuracy column is present. The only accuracy mentioned in Section V-A is 74.52%, and that is attributed to Table IV while the surrounding AUC and MCC values (94.42%, 0.6908) are the final row of Table III, not Table IV. The central claim of the paper is therefore unsupported by the manuscript's own results.","section":"Abstract and Section VI"},{"comment":"The text in Section V-A conflates the two tables: it states 'As shown in Table IV' and then gives validation accuracy of 74.52%, AUROC of 94.42%, and MCC of 0.6908, but Table IV (full-image model) has 10 rows with final ROC AUC 94.41% and MCC 0.7041, while Table III (region-focused model) has 8 rows with final ROC AUC 94.42% and MCC 0.6908. Neither table contains an accuracy value. This internal inconsistency makes it impossible to determine which model achieved the stated 74.52% accuracy, and no definition is given for how validation accuracy was computed.","section":"Section V-A"},{"comment":"The model parameter count is internally inconsistent and does not match a ViT-base architecture. The text reports 597,511 total parameters, with a patch embedding layer of 590,592 parameters and a classification head of 5,383 parameters. These two components sum to 595,975, not 597,511, and neither sum includes the parameters of the transformer encoder, which for google/vit-base-patch16-224-in21k contains approximately 85 million parameters. The reported numbers thus omit the dominant part of the model, calling into question the accuracy of the architectural description.","section":"Section III-B"},{"comment":"The train/test split is not described at the patient level. The NIH ChestX-ray8 dataset contains multiple images per patient, and Section III-A only lists counts of training and test images (12,897 and 2,975). If the split is by image rather than by patient, the same patient's X-rays can appear in both training and test sets, which would inflate accuracy and AUC and invalidate the comparison between the two ViT approaches. The paper should state explicitly whether any patient-level deduplication was performed.","section":"Section III-A"},{"comment":"The abstract and Section IV-C claim that 'both methods surpass the performance of traditional CNN-based models,' but no CNN baseline is trained or evaluated in this study. The only CNN results cited are CheXNet and AutoThorax-Net from other papers using different datasets and evaluation protocols. Without a same-protocol CNN baseline, the claim of superiority over CNNs is not supported by the experiments.","section":"Sections II and IV-C"}],"minor_comments":[{"comment":"The abstract says 'classifcation of diseases on three label' but the dataset described in Section III-A contains seven categories, and the conclusion mentions 'eight' labels. The number of classes should be stated consistently.","section":"Abstract"},{"comment":"Figure 5 is captioned 'Accuracy, Precision, Recall, and F1 Scores' but is never described in the text, and no numeric values for these metrics are given in any table. Either remove the figure or report the corresponding numbers and explain how they were obtained.","section":"Figure 5"},{"comment":"The sentence 'Both approaches outperformed some of the traditional CNN-based models in this study' is unclear, as no CNN models are part of this study.","section":"Section IV-C"},{"comment":"References [10] and [11] are the same ChestX-ray8 paper by Wang et al., listed with different page ranges and venue spellings; one should be removed or the citations should be merged.","section":"References"},{"comment":"There are several typographical errors, including 'classifcation' in the abstract and Section II, 'Comparision' in a table header (if present), and inconsistent capitalization of 'Vision Transformer' and 'AUROC'.","section":"General"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early draft in which the abstract and conclusion report numerical claims that are not present in the experimental results section. The parameter count for the ViT is off by roughly 85 million parameters, which suggests that the architecture description was not verified. The lack of a patient-level split description for NIH ChestX-ray8 is a serious methodological gap. These issues are load-bearing and cannot be addressed by minor revisions; any resubmission would need to report actual accuracy values, correct the architecture details, and add a proper baseline comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know before reading: the headline numbers in the abstract (97.83% and 96.58%) appear nowhere in the results. The only validation accuracy reported is 74.52% for the region-focused model, and Table IV lists AUC and MCC only—no accuracy, precision, recall, or F1. The central claim that a full-image ViT beats CNNs and that segmentation is unnecessary is simply not evidenced.\n\nWhat is actually new here is narrow: the use of SAM2 to segment lung regions and a comparison of full-image versus segmented ViT on a combined NIH ChestX-ray8 + COVID-19 subset. That is an incremental variation, not a new method. The paper does a fair job describing the ViT architecture and training setup, and the question of whether explicit lung segmentation helps is worth asking. But the execution falls apart.\n\nSoft spots, in order of severity. First, the internal inconsistency: the abstract and conclusion report accuracies that the experiments never produce. That alone is fatal. Second, no CNN baseline is trained under the same protocol; comparing to literature AUCs from different datasets is not a valid benchmark. Third, the train/test split is not described at the patient level, and NIH ChestX-ray8 has multiple images per patient—so the reported numbers could easily be inflated by leakage. Fourth, the parameter count is wrong: the stated patch embedding (590,592) plus classification head (5,383) sums to 595,975, not 597,511. Fifth, label counts shift among three, seven, and eight, and Figure 5 is referenced but never explained. Sixth, no code or data is released, so nothing is reproducible.\n\nThe reader's strongest claim holds up: the abstract's figures have no counterpart in the results. The weakest assumption about patient-level splitting is also legitimate, though it is secondary—the primary problem is that the accuracy numbers do not exist in the reported experiments.\n\nWho is this for? Possibly someone wanting a quick example of ViT fine-tuning on chest X-rays, but not a reader looking for reliable evidence. This paper is not ready for peer review; the internal contradiction makes it incoherent on its own terms. If the author can supply the actual accuracy values, run a proper CNN baseline, and describe a patient-wise split, it could become a minor empirical note. As it stands, desk reject.","headline":"The abstract's headline accuracies (97.83%, 96.58%) never appear in the results section; the central claim is unsupported by the paper's own experiments, though the SAM2 segmentation comparison is the only mildly novel element.","tokens_in":7532,"tokens_out":1622,"would_cite":false,"duration_ms":17219,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned Vision Transformer on full chest X-ray images is claimed to reach 97.83% accuracy on three disease labels and 94.54% AUC on eight, beating CNN baselines and a lung-segmented variant.","keywords":["chest X-ray classification","Vision Transformer","lung disease detection","self-attention","medical image classification","transfer learning","lung segmentation","comparative study"],"falsifier":"Take the same images, group them by patient ID from the source dataset, and redo the train/test split so every patient's images are in only one set. If accuracy on the full-image ViT drops substantially below 97.83% or the full-image advantage over the segmented model disappears, the paper's central claim is unsupported. This can be checked without any new imaging data.","tokens_in":6459,"feed_emoji":"🫁","tokens_out":5692,"duration_ms":54608,"temperature":0.7,"pith_summary":"The paper sets out to show that a Vision Transformer fine-tuned for chest X-ray classification can outperform CNN-based models, and that feeding it the full X-ray works better than feeding it lung-segmented regions. The reported evidence is that the full-image ViT reaches up to 97.83% accuracy on three disease labels and 94.54% AUC when the label set expands to eight, while the lung-segmented variant reaches 96.58% accuracy. The author argues that the self-attention mechanism captures global context across the whole image, so explicit lung segmentation is neither necessary nor beneficial. If true, this would simplify preprocessing pipelines for automated chest X-ray diagnosis while improving classification accuracy.","feed_headline":"Full-image ViT hits 97.83% on chest X-rays","feed_subtitle":"No lung segmentation needed; segmented variant trails at 96.58% accuracy.","key_machinery":"The argument runs on the Vision Transformer's self-attention mechanism over image patches, defined as Attention(Q,K,V)=softmax(QK^T/$\\sqrt$(d_k))V, which lets the model weight relationships between distant patches in a 224x224 chest X-ray. The paper contrasts this global-context capacity with the local receptive fields of CNNs, and uses the comparison between full-image and lung-segmented inputs to test whether focusing on the lung region concentrates or degrades the classification signal. The fine-tuned base model is a patch-16 ViT with 12 transformer layers and 597,511 trainable parameters.","core_discovery":"On the paper's own terms, the central discovery is that a standard Vision Transformer with 16x16 patches, fine-tuned on full chest X-ray images, classifies lung diseases more accurately than CNN-based systems and more accurately than the same ViT fine-tuned on lung-segmented images. On the three-label setting the full-image model reaches up to 97.83% accuracy and the segmented model reaches 96.58%; with eight labels the full-image model maintains a 94.54% ROC AUC. The author takes this as evidence that global self-attention can extract disease-relevant features from chest X-rays without explicit lung region segmentation.","pith_inferences":["My inference: the 97.83% and 96.58% accuracy figures are likely computed on different label sets, so they should not be read as a direct head-to-head; the paper should state the exact class configuration for each number.","My inference: re-running the same fine-tuning with a patient-exclusive train/test split would show whether the ViT advantage survives when no patient contributes images to both sets.","My inference: extending the full-image ViT to the complete 14-label set in the source data would test whether the eight-label AUC of 94.54% carries over to a harder multi-label task.","My inference: using the SAM2 masks directly may have injected segmentation boundary noise; a refined mask or attention-based region selection could change the comparison between the two approaches."],"forward_implications":["If the ViT result holds, explicit lung segmentation is unnecessary for accurate chest X-ray classification, simplifying the preprocessing pipeline.","A single fine-tuned ViT can replace CNN backbones while improving accuracy, precision, recall, F1, and AUC in the settings tested.","The eight-label result at 94.54% AUC indicates the method generalizes beyond the three-label setting without architectural changes.","The region-focused variant's lower scores suggest that localization masks do not add signal for a globally attentive model."],"supporting_citations":[{"why":"Supplies the NIH chest X-ray data that the training and test sets are drawn from; the central comparison cannot run without it.","marker":"[11]"},{"why":"Adds the COVID-19 X-ray images that create the COVID-19 class among the seven categories.","marker":"[1]"},{"why":"Introduces the Vision Transformer architecture that the paper fine-tunes for classification.","marker":"[2]"},{"why":"CNN baseline (CheXNet) whose performance the ViT results are compared against.","marker":"[6]"},{"why":"Provides the ChestX-ray8 benchmark and weakly-supervised classification setting that the dataset evaluation builds on.","marker":"[10]"},{"why":"CNN baseline (CheXpert) used as another comparison point for the ViT classification results.","marker":"[3]"},{"why":"Supplies the SAM2 model used to generate the lung-segmented images for the region-focused variant.","marker":"[7]"},{"why":"Defines the self-attention mechanism that the ViT classification head relies on.","marker":"[9]"}],"fun_headline_variants":["ViT beats CNNs on chest X-rays without lung segmentation","Full-image ViT outdoes lung-segmented ViT in X-ray diagnosis","Vision Transformer tops CNN for lung disease detection","97.83% accuracy for ViT on full chest X-rays","No segmentation needed: ViT excels at chest X-ray analysis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison rests on the unstated assumption that no patient appears in both the training and test sets; if the 12,897 training images and 2,975 test images are split without grouping by patient, then the reported accuracy and AUC are inflated and the full-image versus segmented comparison is undermined.","fun_headline_variants_meta":{"raw":{"variants":["ViT beats CNNs on chest X-rays without lung segmentation","Full-image ViT outdoes lung-segmented ViT in X-ray diagnosis","Vision Transformer tops CNN for lung disease detection","97.83% accuracy for ViT on full chest X-rays","No segmentation needed: ViT excels at chest X-ray analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000995,"raw_usage":{"total_tokens":4215,"prompt_tokens":948,"completion_tokens":3267,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":3180}},"tokens_in":564,"tokens_out":3267,"duration_ms":22374,"temperature":1.0,"reasoning_tokens":3180,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:34:38.455563+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same images, group them by patient ID from the source dataset, and redo the train/test split so every patient's images are in only one set. If accuracy on the full-image ViT drops substantially below 97.83% or the full-image advantage over the segmented model disappears, the paper's central claim is unsupported. This can be checked without any new imaging data.","supporting_citations":[{"cited_title":"Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases","cited_arxiv_id":null,"evidence_quote":"Supplies the NIH chest X-ray data that the training and test sets are drawn from; the central comparison cannot run without it."},{"cited_title":"Lungren, and Andrew Y","cited_arxiv_id":null,"evidence_quote":"CNN baseline (CheXNet) whose performance the ViT results are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ChestX-ray8 benchmark and weakly-supervised classification setting that the dataset evaluation builds on."},{"cited_title":"Mong, Safwan S","cited_arxiv_id":null,"evidence_quote":"CNN baseline (CheXpert) used as another comparison point for the ViT classification results."}],"review_version":1}