{"id":"c5944907-650c-4fc7-865b-31571fc275d3","arxiv_id":"2502.04367","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A hybrid ResNet101 plus custom CNN reports 100% test accuracy on a 3,734-image kidney CT test set, but the evaluation uses a portion of the test set for validation, undermining the claim.","lead":"This paper reports a hybrid deep learning model that combines a pre-trained ResNet101 with a custom CNN to classify kidney CT scans into four categories, claiming 100% test accuracy on a public Kaggle dataset. The practical implication is that automated reading of kidney CT images might reach very high accuracy, but the evaluation method in this paper makes the headline number unreliable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test set is not shown to be independent: validation uses test data and the augmentation/split order is ambiguous, so the 100% accuracy claim is unsupported.","rationale":"The paper's central contribution is a single number: 100% test accuracy on 3,734 CT images. For that number to mean anything, the test set must be truly independent of training and model selection. The text explicitly violates this by performing validation on 20% of the test data (Section 3.1). The augmentation/split ordering is also unclear (Section 3.2), and the class counts in Table 2 cannot be reconciled with the reported 3,734-image test set under either ordering. These are not stylistic issues; they directly undermine the falsifiability of the central claim. The architecture description is standard deep learning, and no code, model weights, or reproducibility instructions are provided. The PCA plot itself indicates class overlap between stones and tumors, which makes a perfect test result even more suspicious. The reader's weakest assumption correctly identified this same problem, and I see no reason to soften the rejection. If the authors supply a clean split and the 100% accuracy disappears, the paper should not be accepted as reporting a reliable result.","tokens_in":10243,"tokens_out":3265,"duration_ms":75232,"concrete_test":"Request the exact split and augmentation code from the authors; then run this clean evaluation: split the original 12,446 images into 70% train, 15% validation, 15% test; apply the same augmentation to the training partition only; train the hybrid CNN with early stopping on the validation partition; and report accuracy, precision, recall, F1, and the full confusion matrix on the held-out test partition. If the 100% accuracy is not reproduced on this clean, pre-registered split, the claimed generalization is an artifact of test-set leakage or selection bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of 100% test accuracy depends on the test set being an independent, never-used sample. This is not established. Section 3.1 states 'Validation is performed on 20% of the testing data', meaning a subset of the 3,734 test images was used for model selection or early stopping; the reported test metric is therefore not a fresh evaluation. Section 3.2 describes augmentation without stating whether it was applied before or after the 70/30 split. The reported test set size, 3,734, is 30% of the original 12,446 images, not 30% of the augmented 27,035 images. If augmentation preceded the split, the test set should contain roughly 8,110 augmented images; if it followed the split, the training set cannot be reconciled with the augmented class counts in Table 2. In either reading, the paper does not demonstrate that the 3,734 test images were held out from all training and model-selection steps. Additionally, the architecture has 15.6 million trainable parameters and was trained for only 5 epochs on roughly 8,700 original training images, while the PCA analysis in Section 6.2.2 shows tumor and stone clusters overlapping; a perfect zero-error result on all four classes is surprising and requires stronger evidence than a one-line claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a hybrid deep learning architecture that combines a pre-trained ResNet101 with a custom CNN, using feature fusion (an 'IntersectFeatures' operation) to classify kidney CT images into normal, stone, cyst, and tumor. The central claim is that this hybrid model achieves 99.73% training accuracy and 100% test accuracy, with a perfect confusion matrix on a 3,734-image test set, outperforming a standalone ResNet101 and a base CNN. The paper reports hyperparameters, per-epoch training metrics, confusion matrices, and PCA visualizations.","tokens_in":10506,"tokens_out":3043,"duration_ms":31193,"significance":"If the reported result were valid, a perfect four-class classification of kidney CT images would be a practically significant outcome, and the dual-branch ResNet101/custom-CNN fusion idea would merit attention. The paper uses a public dataset and provides a fairly detailed architecture table and hyperparameter list. However, the evaluation protocol described in Section 3.1 explicitly uses test data for validation, and Section 3.2 leaves the augmentation/split order ambiguous. These issues are not cosmetic: they directly undermine the independence of the test set and therefore the paper's headline claim. The significance of the paper is currently contingent on an evaluation that is not demonstrably sound.","major_comments":[{"comment":"The validation protocol invalidates the test result as an independent measurement. Section 3.1 states 'Validation is performed on 20% of the testing data,' meaning that a subset of the 3,734 test images was used for model selection, early stopping, or hyperparameter choices during training. The reported 100% test accuracy in Table 6 and the Abstract is therefore not a fresh evaluation on never-before-used data. Since the number of epochs (5) and other choices could have been tuned using the validation split drawn from the test set, the central claim of perfect generalization is unsupported.","section":"§3.1, §6.2, Abstract"},{"comment":"The relationship between the augmented dataset and the train/test split is ambiguous. The test set size is reported as 3,734 images, which equals 30% of the original 12,446 images, not 30% of the augmented 27,035 images listed in Table 2. The paper does not state whether augmentation was applied before or after the 70/30 split. If augmentation preceded the split, the test set should contain roughly 8,110 images; if augmentation followed the split, the per-class counts in Table 2 cannot be reconciled with a 70/30 split of the original counts. This ambiguity leaves open the possibility that augmented copies of training images appear in the test set, which would inflate the reported accuracy.","section":"§3.2, Table 2"},{"comment":"There is an internal contradiction and a tension with the claimed perfect classification. The text says 'Figure 6 shows the confusion matrix of the ResNet101 model,' but Figure 6 is subsequently described as PCA scatter plots. More importantly, the PCA visualization shows overlapping tumor and stone clusters, which is at odds with the hybrid CNN's reported perfect confusion matrix (zero errors on all four classes). The paper should either reconcile these observations or provide a quantitative explanation of how the feature fusion resolves the overlap.","section":"§6.2.2, Figure 6"},{"comment":"The training-time numbers are inconsistent. Section 5.1 states the model required 132 seconds per epoch, and Table 4 specifies 5 epochs; this amounts to about 11 minutes of training. Table 6, however, reports a hybrid CNN training time of 1:25:02 (85 minutes). Additionally, Table 5 reports 100% precision, recall, and F1 from Epoch 3 onward while accuracy is below 100% (99.45% and 99.73% in Epochs 4 and 5), which is arithmetically possible only for per-class macro-averaged metrics; the paper should report per-class metrics and state the averaging scheme.","section":"§5.1, Table 5, Table 6"}],"minor_comments":[{"comment":"The 'IntersectFeatures' operation is never defined mathematically; Algorithm 1 says 'Keep the features extracted from both ResNet101 and custom CNN with higher similarity,' but the similarity criterion, threshold, and feature-alignment mechanism are unspecified. A precise definition is needed for reproducibility.","section":"§4, Algorithm 1"},{"comment":"The opening sentence 'Figure 6 shows the confusion matrix of the ResNet101 model' appears to be a leftover from an earlier draft; Figure 6 is captioned as PCA plots. This should be corrected.","section":"§6.2.2"},{"comment":"The column header 'Number of Kindly Images' contains a typo; it should read 'Number of Kidney Images.'","section":"Table 2"},{"comment":"The caption of Table 5 says the table provides a classification report over five epochs, but the reported standard deviation values are not defined in the text; please clarify what quantity the standard deviation is computed over.","section":"§5.3, Table 5"},{"comment":"The text in Section 2 attributes a precision of 99.88% to Sharma et al. (2024), but Table 1 lists a 96.52% accuracy for that work; the relationship between these numbers should be clarified.","section":"§2, Table 1"}],"recommendation":"reject","confidential_remarks":"The core problem is the evaluation design: using test data for validation is a fundamental methodological error, and the augmentation/split ambiguity makes it impossible to rule out data leakage. These are load-bearing issues, not presentation issues; fixing them would require a complete re-run of the experiments with a properly held-out test set. Under the journal's standards, this warrants rejection. The paper does contain a reproducible public dataset and transparent hyperparameters, but the reported performance cannot be taken at face value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is not a paper I'd send to review. The headline number is unsupported by the protocol as written. Section 3.1 says validation is performed on 20% of the testing data, which means the test set is no longer independent. The epoch count and early stopping are chosen using labels from the test split, so the reported 100% accuracy is partly fitted to the evaluation data. That alone is a load-bearing flaw.\n\nTo be fair, the paper does some things well. The architecture is described in detail with a layer-by-layer table, the authors compare against two baselines, and they report per-epoch metrics, a confusion matrix, and PCA. The dataset is public, so others can reproduce the split.\n\nThe softer spots: the split/augmentation order is ambiguous. The test set size (3,734) is 30% of the original 12,446, which suggests augmentation was applied only to the training portion, but the paper never says so. The IntersectFeatures operation is never defined mathematically; it's a black box that \"filters common features with higher similarity,\" which makes the architecture hard to assess or reproduce. Novelty is modest—feature fusion between a pretrained CNN and a custom CNN already appears in the cited literature, including Hossain et al. 2023b and Ali et al. 2023. The only genuinely new ingredient is the specific ResNet101 + custom CNN combination, and that's incremental.\n\nI also note the PCA in Section 6.2.2 shows tumor and stone clusters overlapping, which makes a perfect zero-error test result surprising. That's not proof of error, but it raises the bar for evidence, and the paper doesn't meet it.\n\nBottom line: the central claim is not supported by the evaluation protocol. The paper is not incoherent; it's just that the design of the experiment invalidates the headline. A serious referee would catch this immediately, but I don't think the manuscript deserves a full review cycle. Desk reject is appropriate. If you want a short cautionary example for a methods class, it works—but as a research contribution, it doesn't.","headline":"The paper's 100% test accuracy is not credible because the test set is used for validation; the rest is a routine feature-fusion baseline on a public dataset.","tokens_in":11052,"tokens_out":2850,"would_cite":false,"duration_ms":42281,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid ResNet101-plus-custom-CNN architecture classifies kidney CT images into normal, stone, cyst, and tumor categories, reporting 100% test accuracy on a 3,734-image test set.","keywords":["kidney CT classification","hybrid CNN","ResNet101","feature fusion","kidney stones","kidney cysts","kidney tumors","data augmentation"],"falsifier":"Rerun the pipeline with the 70/30 split made before any augmentation, augment only the training portion, and measure accuracy on the untouched held-out images; if accuracy falls below 100% or errors appear on stones or tumors, the reported perfect score depended on test contamination or validation leakage. A second check is to test the trained model on an external kidney CT dataset with the same four labels.","tokens_in":10023,"feed_emoji":"🩻","tokens_out":8929,"duration_ms":65772,"temperature":0.7,"pith_summary":"The paper tries to establish that fusing a pre-trained ResNet101 with a custom CNN through feature fusion can classify kidney CT images into four clinically distinct categories with perfect accuracy on its test set. If true, an automated system could flag stones, cysts, and tumors from CT scans with no false positives or false negatives in this dataset, and do so with a faster testing time than ResNet101 alone. The architecture splits the task into normal-vs-stone and cyst-vs-tumor branches inside ResNet101, then merges those features with a custom CNN stream via an IntersectFeatures operation that keeps high-similarity features. The authors report 99.73% training accuracy and 100% test accuracy, precision, recall, and F1 on a 3,734-image test drawn from a 12,446-image public CT dataset expanded by augmentation to 27,035 images.","feed_headline":"Hybrid CNN hits 100 percent on kidney CT test","feed_subtitle":"A fused ResNet101-plus-CNN classifies 3,734 kidney CT images with zero errors, beating standalone models.","key_machinery":"The load-bearing component is feature fusion implemented by an IntersectFeatures operation, which combines the features extracted from a pre-trained ResNet101 and a custom CNN and keeps the features with higher similarity between the two streams. The fused representation is then flattened and passed through dense layers with dropout and a softmax head that outputs probabilities for normal, stone, cyst, and tumor. A second design choice is the dual-branch split of ResNet101, with one branch specializing in normal-vs-stone discrimination and the other in cyst-vs-tumor discrimination, so that the features merged into the final classifier encode task-specific knowledge from both subtasks.","core_discovery":"The central claim is that the hybrid model assigns every image in its 3,734-image test set to the correct class, achieving 100% accuracy, precision, recall, and F1, while ResNet101 alone reaches 99.81% accuracy and the base CNN only 77.39%. In the confusion matrix the hybrid model has zero misclassifications, including for stones, a class that ResNet101 confuses with cysts in four cases. The authors interpret this as evidence that fusing the two feature streams filters out common or noisy features and separates class clusters that overlap in the ResNet101 feature space, as illustrated by PCA plots showing cleaner separation for the hybrid model.","pith_inferences":["Beyond the paper: the perfect score is best understood as dataset-level, since the paper does not fully specify whether augmentation occurred before or after the 70/30 split; applying the same pipeline to an external CT dataset would test whether the fused features generalize.","Beyond the paper: the dual-branch design encodes the clinical prior that normal-vs-stone and cyst-vs-tumor are separable subtasks; ablating one branch would reveal how much of the gain comes from this split rather than from feature fusion.","Beyond the paper: the IntersectFeatures operation, which retains high-similarity features from both streams, resembles a consensus filter and could transfer to other small-data medical imaging tasks if it is the active ingredient.","Beyond the paper: the unusually fast convergence and perfect validation scores from epoch 3 onward suggest that the reported accuracy should be rechecked on a strictly separated test set before clinical deployment."],"forward_implications":["A fused two-stream CNN can separate all four kidney classes in this dataset, including the stone/tumor pairs that ResNet101 alone confuses.","The 100% per-class recall means the model misses no tumors or stones in the 3,734-image test set, which is the outcome that matters for screening.","The model reaches near-perfect accuracy within five epochs, so the fused representation is learned quickly on well-annotated CT data.","Testing time of 23 seconds for 3,734 images is shorter than ResNet101's 31 seconds, supporting use in high-throughput reading.","The dual-branch design shows that encoding clinical subtasks explicitly can improve a transfer-learning baseline on imbalanced medical image classes."],"supporting_citations":[{"why":"Supplies the closest architectural precedent: a customized CNN combined with ResNet50 that reached 98.66% accuracy, which this paper extends to ResNet101 with dual-branch fusion.","marker":"Hossain et al. (2023b)"},{"why":"Source of the feature-fusion idea for kidney abnormality detection, showing that combining multiple neural networks improves identification of renal pathologies.","marker":"Wu and Yi (2020)"},{"why":"Establishes the same class distribution and a 96.52% InceptionV3 baseline that the hybrid model is designed to outperform.","marker":"Sharma et al. (2024)"},{"why":"Provides a VGG16 transfer-learning baseline with 99.7% training and validation accuracy, setting the performance level the hybrid model must match.","marker":"Prasher et al. (2024)"},{"why":"Gives a CNN/SVM mutual-learning baseline on the same data distribution whose lower accuracy motivates the hybrid architecture.","marker":"Chowdhury et al. (2023)"},{"why":"Contributes the DeepKidney CNN baseline for multi-class kidney classification across stones, cysts, and tumors.","marker":"Bindu Madavi and Sowjanya (2023)"}],"fun_headline_variants":["Fused CNN nails 100% on kidney CT scans","ResNet+CNN fusion hits perfect score on kidney CT","Hybrid model achieves 100% accuracy on kidney CT test","Zero errors for fused ResNet-CNN on kidney CT","Kidney CT: fused ResNet-CNN gets stones, cysts, tumors right"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The perfect test score rests on the test images being truly unseen during training and model selection, but the paper does not state clearly that augmentation happened only after the 70/30 split and it uses 20% of the test set for validation.","fun_headline_variants_meta":{"raw":{"variants":["Fused CNN nails 100% on kidney CT scans","ResNet+CNN fusion hits perfect score on kidney CT","Hybrid model achieves 100% accuracy on kidney CT test","Zero errors for fused ResNet-CNN on kidney CT","Kidney CT: fused ResNet-CNN gets stones, cysts, tumors right"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000873,"raw_usage":{"total_tokens":3724,"prompt_tokens":834,"completion_tokens":2890,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":2803}},"tokens_in":450,"tokens_out":2890,"duration_ms":17787,"temperature":1.0,"reasoning_tokens":2803,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:19:58.414464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the pipeline with the 70/30 split made before any augmentation, augment only the training portion, and measure accuracy on the untouched held-out images; if accuracy falls below 100% or errors appear on stones or tumors, the reported perfect score depended on test contamination or validation leakage. A second check is to test the trained model on an external kidney CT dataset with the same four labels.","supporting_citations":[{"cited_title":", author Pokhariya, H.S","cited_arxiv_id":null,"evidence_quote":"Establishes the same class distribution and a 96.52% InceptionV3 baseline that the hybrid model is designed to outperform."},{"cited_title":", author Nelson, L","cited_arxiv_id":null,"evidence_quote":"Provides a VGG16 transfer-learning baseline with 99.7% training and validation accuracy, setting the performance level the hybrid model must match."},{"cited_title":", author Mukhopadhyay, S","cited_arxiv_id":null,"evidence_quote":"Gives a CNN/SVM mutual-learning baseline on the same data distribution whose lower accuracy motivates the hybrid architecture."},{"cited_title":", author Sowjanya, K","cited_arxiv_id":null,"evidence_quote":"Contributes the DeepKidney CNN baseline for multi-class kidney classification across stones, cysts, and tumors."}],"review_version":1}