{"id":"d325f82c-b520-4f7d-97b0-60f7c6fcd605","arxiv_id":"2508.13461","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A ViT-base model pretrained on ImageNet-21k outperforms a ResNet50 baseline on kidney stone subtype classification, but the advantage may come from pretraining scale, not the transformer architecture.","lead":"The paper compares Vision Transformer and CNN models for classifying kidney stone subtypes from endoscopic images on two ex vivo datasets. A ViT pretrained on ImageNet-21k beats a ResNet50 baseline by large margins, but the results confound architecture with pretraining scale.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main comparison confounds architecture with pretraining scale: at matched ImageNet-1k pretraining, ViT loses to ResNet50 on five of six subsets, so 'superiority' rests on an untested ResNet50-21k baseline.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the missing ImageNet-21k CNN baseline makes the ViT-vs-CNN comparison confounded by pretraining scale. The paper's own Table 3 shows that at matched ImageNet-1k pretraining, ViT loses to ResNet50 on five of six subsets, which directly undercuts the overgeneralized conclusion that ViT-based architectures are superior. The correct fix is a controlled baseline experiment, not a rejection of the underlying data or task. The reader's CONDITIONAL verdict already reflects this, so I do not recommend changing it. I would keep the condition explicit: the authors must add a 21k-pretrained CNN baseline and temper the abstract/conclusion claims if the baseline closes the gap. No code or detailed training sections were available in the extract, but the specific missing baseline is sufficient to justify the conditional verdict.","tokens_in":7262,"tokens_out":2765,"duration_ms":29432,"concrete_test":"Train ResNet50 and ResNet152 with ImageNet-21k/22k pretrained weights (e.g., timm checkpoints) using the same training protocol, augmentation, and evaluation splits as the ViT-21k models, and report all six subset accuracies/F1. If ResNet50-21k matches or exceeds ViT-21k on at least half of the subsets, the paper's architecture-level conclusion collapses; if ViT-21k still wins across the board, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that ViT-based architectures are superior for kidney stone classification—is supported in Table 3 only by comparing ViT-base pretrained on ImageNet-21k against ResNet50 pretrained on ImageNet-1k. At matched ImageNet-1k pretraining, ViT-base underperforms ResNet50 on five of six subsets: A MIX 0.624 vs 0.784, A SEC 0.657 vs 0.809, A SUR 0.492 vs 0.796, B MIX 0.769 vs 0.850, B SEC 0.739 vs 0.645 (only ViT win), B SUR 0.704 vs 0.864. The abstract's 'consistently outperformed' is also internally contradicted by A SUR, where ResNet50-1k (0.796) beats ViT-21k (0.768). The paper never trains a CNN on ImageNet-21k, so the architecture comparison is confounded by pretraining data scale. If a ResNet50-21k baseline matched or exceeded ViT-21k, the conclusion would reduce to a pretraining-data story rather than a transformer advantage. This is an external-validity problem, not an internal inconsistency, but it is the load-bearing gap in the paper's strongest claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares Vision Transformer (ViT) and CNN (ResNet) models for kidney stone subtype classification on two ex vivo image datasets, one from CCD cameras and one from flexible ureteroscopes. Across six view subsets, the authors report accuracy, F1, precision, and recall for ResNet50/ResNet152 and ViT-base/ViT-small under several pretraining conditions (ImageNet-1k, ImageNet-21k, from scratch), together with feature-embedding analyses based on t-SNE, silhouette scores, and intra-class distances. The central claim is that ViT architectures—particularly ViT-base pretrained on ImageNet-21k—provide superior classification performance and more discriminative embeddings than CNNs.","tokens_in":7528,"tokens_out":7507,"duration_ms":78074,"significance":"If the claim were established under controlled conditions, the result would be practically relevant for automated endoscopic kidney stone recognition, and the paper's combination of a multi-subset benchmark with an ablation over pretraining is a useful empirical contribution. I see no circularity: the headline numbers are measured outcomes on held-out test images. The main weakness is that the architecture comparison is confounded with pretraining data scale: the paper compares ViT pretrained on ImageNet-21k against ResNet50 pretrained on ImageNet-1k, and its own matched-pretraining rows largely contradict the superiority claim. The paper also reports headline abstract numbers that differ from the corresponding entries in Table 3. The strengths are the breadth of the benchmark and the inclusion of several quantitative embedding diagnostics, but the central comparative conclusion needs additional controlled experiments or a substantially more cautious framing.","major_comments":[{"comment":"The paper's central claim—that ViT architectures provide superior classification performance (Sec. 7)—rests on comparing ViT-base pretrained on ImageNet-21k with ResNet50 pretrained on ImageNet-1k. This confounds architecture with pretraining data scale. In the matched ImageNet-1k rows, ViT-base is worse than ResNet50 on five of six subsets (A MIX 0.624 vs 0.784; A SEC 0.657 vs 0.809; A SUR 0.492 vs 0.796; B MIX 0.769 vs 0.850; B SUR 0.704 vs 0.864), and better only on B SEC (0.739 vs 0.645). An ImageNet-21k-pretrained CNN baseline (e.g., ResNet50-21k) is needed before attributing the improvement to the transformer architecture; otherwise the results support a pretraining-scale effect, not an architecture effect.","section":"Table 3"},{"comment":"The headline examples in the abstract do not match the reported table. The abstract states 87.1% accuracy for the CCD mixed-view subset and 95.2% accuracy/95.1% F1 for the endoscopic section subset, but Table 3 reports 0.890 for A MIX (ViT-base/21k) and 0.957/0.958 for B SEC. If these figures come from a different configuration or table, that configuration must be cited; as printed, the key numbers are not reproducible from the paper's main table.","section":"Abstract vs. Table 3"},{"comment":"Even under the authors' chosen (unmatched) comparison, the statement that ViT 'consistently outperformed' ResNet50 is contradicted by the A SUR row: ResNet50/ImageNet-1k achieves 0.796 accuracy vs 0.768 for ViT-base/ImageNet-21k. The conclusions should be qualified to name the subsets and pretraining conditions for which ViT wins, and should not claim general superiority.","section":"Abstract and Sec. 7"},{"comment":"The embedding comparison ('same experimental conditions') is also affected by the pretraining confound. If the ViT features come from the ImageNet-21k model and the CNN features from the ImageNet-1k model, the higher silhouette scores and lower intra-class distances may reflect pretraining scale rather than architecture. Please specify which rows of Table 3 are used, and either repeat the embedding analysis with matched pretraining or discuss the confound explicitly.","section":"Sec. 6.2, Fig. 5, Table 4"}],"minor_comments":[{"comment":"The statement that ViTs 'require less data for training because their self-attention mechanism efficiently captures global contextual relationships' is not supported and appears contradicted by Table 3: from-scratch ViT-base attains 0.209--0.505 across subsets, whereas from-scratch ResNet50 attains 0.583--0.773. Please remove or substantiate this claim.","section":"Sec. 1.2"},{"comment":"To assess whether the reported differences are stable, please add confidence intervals, repeated-run variance, or at least the number of test images per subset. The current point estimates, especially for small subsets, do not by themselves establish 'consistent' superiority.","section":"Table 3"},{"comment":"The table header appears garbled (e.g., 'Subv敲獩潮'). Please ensure the final PDF typesets the header correctly.","section":"Table 3 header"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the unfair comparison: the main table shows that at matched ImageNet-1k pretraining the ViT is not superior, so the paper's framing depends entirely on an untested ImageNet-21k CNN baseline. Adding such a baseline (or an equivalent controlled comparison) is within the scope of a revision and should be the primary condition. The abstract/table numerical mismatch also needs correction before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a straightforward benchmark: stock ViT-Base and ResNet50/152 on two ex vivo kidney stone datasets, with an ablation over pretraining (ImageNet-1k, ImageNet-21k, none) and model size. What's new is the specific result that ViT-Base pretrained on ImageNet-21k beats ResNet50 pretrained on ImageNet-1k on most subsets of these datasets. That is a real measured benchmark, and Table 3 is honestly laid out with all the numbers visible.\n\nThe soft spot is the conclusion. At matched ImageNet-1k pretraining, ViT loses to ResNet50 on five of six subsets. The paper never trains a CNN on ImageNet-21k. So the headline 'ViT-based architectures provide superior classification performance' is not supported by the data; what's supported is that a 21k-pretrained ViT beats a 1k-pretrained ResNet. That is likely a pretraining-scale effect. The abstract's 'consistently outperformed' is also internally contradicted by A-SUR, where ResNet50-1k beats ViT-21k. The stress-test note is right.\n\nOther concerns are minor: no error bars, single runs, no code/data artifacts. Section 6.3 selects display samples by the gap it claims to demonstrate, which is circular for qualitative evidence. Sections 3-5 are not in the extract I reviewed, so I couldn't verify training details, but the Table 3 numbers are self-consistent with the protocol described.\n\nThe paper does earn credit for the systematic ablation table and the embedding/intra-class distance analysis, which is more than a first-to-apply claim.\n\nBottom line: this is a useful dataset-specific benchmark, but the central claim needs a matched ResNet50-21k baseline (and ideally error bars) before the conclusion can stand. I'd send it to peer review, not desk reject, because the comparison is well-defined and the missing baseline is an addressable flaw. I wouldn't cite the superiority claim as it stands, but I'd use Table 3 as an example of pretraining scale confounding architecture comparisons.","headline":"Useful benchmark, but the ViT superiority claim is a pretraining-scale artifact until a ResNet50-21k baseline is added.","tokens_in":8137,"tokens_out":1735,"would_cite":false,"duration_ms":16305,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a Vision Transformer pretrained on ImageNet-21k classifies kidney stone subtypes from endoscopic images more accurately than a ResNet50 CNN pretrained on ImageNet-1k, with the largest margin on the hardest section-patc","keywords":["Vision Transformer","kidney stone classification","endoscopic imaging","CNN baseline","ImageNet-21k pretraining","self-attention","morpho-constitutional analysis","ex vivo datasets"],"falsifier":"Run the identical split and training protocol with a ResNet50 pretrained on ImageNet-21k (or both models initialized from the same 21k checkpoint) and compare on all six dataset-by-view subsets. If the 21k-pretrained CNN matches or exceeds the ViT, the architectural conclusion is falsified and the observed gap was pretraining data, not self-attention.","tokens_in":7089,"feed_emoji":"🪨","tokens_out":10593,"duration_ms":104464,"temperature":0.7,"pith_summary":"The paper's central claim is that a standard Vision Transformer (ViT-base) pretrained on ImageNet-21k classifies kidney stone subtypes from endoscopic images more accurately than a ResNet50 CNN pretrained on ImageNet-1k. On the hardest subset—section patches from ureteroscope images—the reported numbers are 95.7% versus 64.5% accuracy, and on mixed-view CCD images 87.1% versus 78.4%. The authors attribute the advantage to self-attention's global patch dependencies and to tighter, more separable feature embeddings. If correct, automated visual stone recognition during ureteroscopy becomes a scalable route to personalized treatment and recurrence prevention.","feed_headline":"Transformer model beats CNN on kidney stone images: 96% vs 65%","feed_subtitle":"A 21k-pretrained ViT hits 95.7% on hard endoscopic patches; ResNet50 scores 64.5%.","key_machinery":"The central object is the ViT-base model with the standard patch-and-self-attention pipeline: the image is cut into fixed-size patches, each patch is linearly embedded, and multi-head self-attention re-weights every patch against every other patch before classification. It supplies global context in a single pass, which the paper says is what lets the model form compact, well-separated clusters under lighting and view variability. The ImageNet-21k pretraining is a second, equally load-bearing component: it is the version that yields the headline numbers, while the same ViT pretrained on ImageNet-1k or from scratch performs much worse.","core_discovery":"The authors' positive claim is that transformer-based models, specifically ViT-base pretrained on ImageNet-21k, classify kidney stone subtypes from endoscopic images more accurately and with better feature structure than the ResNet50 CNN baseline. The support consists of an ablation across two datasets and three view types, embedding visualizations with silhouette scores, intra-class distance tables, confusion matrices, and a qualitative sample comparison. The authors attribute the results to self-attention's global patch-level context, which lets the model separate visually ambiguous stone subtypes that trip up convolutional features.","pith_inferences":["Editorial inference: the headline gap is as much about pretraining scale as about architecture. ViT-base pretrained on ImageNet-1k trails ResNet50-ImageNet-1k on five of the six subsets in Table 3, so the 21k pretraining is doing consequential work.","Editorial inference: the direct falsifying experiment is to train a ResNet50 with ImageNet-21k pretraining under the same protocol and compare; if the CNN matches or exceeds the ViT, the observed advantage is a pretraining-data effect rather than a self-attention effect.","Editorial inference: the paper's 'consistently outperformed' phrasing is stronger than the table shows—on Dataset A surface views the ViT-21k scores 76.8% versus ResNet50's 79.6%—so the claim depends on which subsets are highlighted."],"forward_implications":["Automated stone subtype recognition could be built around a single pretrained ViT rather than task-specific CNN pipelines.","The large margins on hard Section patches suggest transformer features hold up where lighting and view artifacts break CNN embeddings.","Compact ViT embeddings, with lower intra-class distances and higher silhouette scores, imply better retrievability of visually similar stone subtypes.","If deployed, the system could give urologists an immediate visual read during ureteroscopy, reducing the need to extract fragments for infrared laboratory analysis."],"supporting_citations":[{"why":"Supplies Dataset A (CCD-camera images) and the stone-type classes used in the comparisons.","marker":"[2]"},{"why":"Supplies the ViT architecture and the ImageNet-21k pretraining recipe that produces the best-performing model.","marker":"[5]"},{"why":"Supplies Dataset B (flexible ureteroscope images), including the Section-patch subset with the largest ViT lead.","marker":"[6]"},{"why":"Defines Endoscopic Stone Recognition, the clinical task the classifier is meant to automate.","marker":"[8]"},{"why":"Prior machine-learning work on in vivo kidney-stone recognition that this study extends and compares against.","marker":"[14]"},{"why":"The prior CNN transfer-learning pipeline that provides the ResNet baseline context.","marker":"[15]"}],"fun_headline_variants":["ViT beats CNN on hardest stone patches: 95% vs 65%","Transformers outclass CNNs for kidney stone classification","ImageNet-21k pretrained ViT tops ResNet50 on stones","ViT hits 95% accuracy on hard stone images, CNN only 65%","Self-attention beats convolutions for kidney stone recognition"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The central comparison is not architecture-only: the ViT uses ImageNet-21k pretraining while the ResNet uses ImageNet-1k, and on one surface-view subset the ResNet already leads; the paper's conclusion assumes that pretraining scale is not doing most of the work.","fun_headline_variants_meta":{"raw":{"variants":["ViT beats CNN on hardest stone patches: 95% vs 65%","Transformers outclass CNNs for kidney stone classification","ImageNet-21k pretrained ViT tops ResNet50 on stones","ViT hits 95% accuracy on hard stone images, CNN only 65%","Self-attention beats convolutions for kidney stone recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000986,"raw_usage":{"total_tokens":4003,"prompt_tokens":711,"completion_tokens":3292,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":3198}},"tokens_in":455,"tokens_out":3292,"duration_ms":24508,"temperature":1.0,"reasoning_tokens":3198,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T19:01:53.291727+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical split and training protocol with a ResNet50 pretrained on ImageNet-21k (or both models initialized from the same 21k checkpoint) and compare on all six dataset-by-view subsets. If the 21k-pretrained CNN matches or exceeds the ViT, the architectural conclusion is falsified and the observed gap was pretraining data, not self-attention.","supporting_citations":[{"cited_title":"European Urology Focus 7(1), 13–21 (2021) 2, 4, 5, 8 Kidney Stone Recognition with ViT 17","cited_arxiv_id":null,"evidence_quote":"Supplies Dataset A (CCD-camera images) and the stone-type classes used in the comparisons."},{"cited_title":"BJU international130(6), 786–798 (2022) 4, 5, 8","cited_arxiv_id":null,"evidence_quote":"Supplies Dataset B (flexible ureteroscope images), including the Section-patch subset with the largest ViT lead."},{"cited_title":"PROGRES EN UROLO- GIE 27(2), F26–F35 (2017) 1, 3","cited_arxiv_id":null,"evidence_quote":"Defines Endoscopic Stone Recognition, the clinical task the classifier is meant to automate."},{"cited_title":"IEEE Access12, 10736–10759 (2024) 3, 6","cited_arxiv_id":null,"evidence_quote":"Prior machine-learning work on in vivo kidney-stone recognition that this study extends and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior CNN transfer-learning pipeline that provides the ResNet baseline context."}],"review_version":1}