{"id":"3c1882c3-8fd5-4583-b9d5-d77dca9cf44a","arxiv_id":"2508.20877","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"A ResNet-18 model trained on dual-modality microscopy images distinguishes pancreatic cancer from non-cancer with reported 96.4% sensitivity and specificity, but the evaluation is weakened by small sample size and likely patient-level leakage.","lead":"This paper trains deep learning models on paired autofluorescence and SHG microscope images of pancreatic tissue from 40 patients to classify tissue as normal, fibrotic, or cancerous. It reports over 90% accuracy in detecting pancreatic cancer, but the evaluation has serious methodological gaps that make the result uncertain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Patient-level leakage likely inflates binary accuracy: 5-fold CV is not described as grouped by patient, and multiple images per patient make cross-contamination probable.","rationale":"The reader's weakest assumption—that cross-validation is not patient-grouped—is exactly the most load-bearing concern. The paper's central quantitative claim is a near-perfect binary classification result obtained from a small, highly clustered dataset. The text provides no evidence that images from the same patient were kept together during splitting, and the dataset description strongly implies multiple images per patient. Since the entire clinical claim depends on generalizing to unseen patients, an ungrouped split would invalidate the reported metrics. I agree with the reader's REJECT verdict: as written, the evidence does not support the claim of 96.4% sensitivity/specificity for new patients. The concern is concrete and testable via the public repository or a rerun with grouped cross-validation.","tokens_in":10182,"tokens_out":2072,"duration_ms":23950,"concrete_test":"Inspect the public code repository (https://github.com/deslo-research/pdac_ml) for the exact split implementation used for the binary classifier. If the split is random by image, rerun the final ResNet-18 binary model using GroupKFold(n_splits=5) with patient ID as the grouping key, keeping all images from one patient in the same fold. Compare the patient-exclusive AUC, sensitivity, and specificity against the reported 0.989/0.964/0.964, with bootstrap confidence intervals. Also aggregate predictions at the patient level (e.g., majority vote over a patient's images) and report patient-level accuracy. If patient-exclusive AUC drops substantially below 0.989, the headline result is inflated by leakage.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim of 0.964 sensitivity/specificity and 0.989 AUC rests on a 5-fold cross-validation over 239 images from only 40 patients. Section 2.1 states that eight ductal structures were imaged per tissue category, so each patient contributes multiple images. Section 2.6 describes the k-fold procedure only as 'dividing the dataset'; it never states that all images from a given patient are kept within the same fold. If splits are random at the image level, each validation fold will contain images from patients whose other images appear in the training folds, allowing the model to exploit patient-specific acquisition artifacts, staining properties, or biological idiosyncrasies that would not generalize to new patients. With 40 patients and 5 folds, a random image-level split means roughly 32 of 40 patients appear in both train and validation for a given fold, so near-perfect AUC is exactly what leakage would produce. This is the load-bearing assumption for the headline result: if it fails, the reported sensitivity, specificity, and AUC are not estimates of generalization to new patients. The paper's other inconsistencies—the unexplained initial validation accuracy of 13%, the conflicting sensitivity values (0.964 vs. 92.3% from Figure 13), and the absence of confidence intervals—reinforce but do not replace this primary concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a deep learning framework to classify pancreatic tissue images into normal, fibrotic, and cancerous categories using combined autofluorescence and second harmonic generation (SHG) microscopy. The authors compare six CNN/ViT architectures, adopt transfer learning with frozen backbones to combat overfitting, apply class-weighted training for imbalance, and use 5-fold cross-validation. They then build a binary ResNet-18 classifier for cancer vs. non-cancer, reporting sensitivity and specificity of 0.964 and ROC AUC of 0.989. The manuscript includes Grad-CAM interpretability analysis and makes code publicly available. The central claim is that this framework achieves near-perfect classification performance and has potential for clinical deployment in early PDAC detection.","tokens_in":10547,"tokens_out":7164,"duration_ms":75523,"significance":"If the reported results are reliable, this would be a clinically valuable contribution: an automated, interpretable pipeline for PDAC tissue classification using optical imaging, with strong sensitivity and specificity on a limited dataset. The paper's strengths include a systematic comparison of architectures, explicit handling of class imbalance, public code release, and interpretability analysis. However, the evaluation protocol has a potentially fatal methodological gap: the cross-validation is not described as patient-grouped, despite multiple images per patient, so the headline metrics may reflect patient-specific leakage rather than generalization to new patients. Additional internal inconsistencies further undermine confidence. The significance is therefore conditional on a rigorous re-analysis.","major_comments":[{"comment":"The dataset contains multiple images per patient (Section 2.1: eight ductal structures per tissue category; 239 images from 40 patients), yet the 5-fold cross-validation in Section 2.6 is described only as 'dividing the dataset' into five portions. No patient-level grouping is mentioned. If images from the same patient appear in both training and validation folds, the model can exploit patient-specific artifacts or biological idiosyncrasies, artificially inflating sensitivity, specificity, and AUC. This is the load-bearing assumption for the central claim. The authors must specify whether folds are stratified by patient and, if not, re-run the evaluation with patient-independent splits and report the resulting metrics.","section":"Section 2.6 (with Section 2.1)"},{"comment":"The abstract states that the final optimized framework uses 'frozen pre-trained layers', but Section 3.4.1 describes the final binary implementation as a 'fully trainable network' with dropout and a 512-unit hidden layer. This direct contradiction affects reproducibility and the description of the method. Please clarify which configuration produced the reported binary classification results, and ensure the abstract matches the actual final architecture.","section":"Abstract vs. Section 3.4.1"},{"comment":"The text claims 'both sensitivity and specificity reaching 0.964', but the confusion matrix in Figure 13 shows cancer detection at 24/26 (92.3%) and non-cancer detection at 27/28 (96.4%). The cancer sensitivity is 92.3%, not 96.4%. The Conclusion repeats '96.4% sensitivity and specificity'. This is a numerical inconsistency in the headline result; it must be reconciled or corrected.","section":"Section 3.4.2 and Figure 13"},{"comment":"The paper reports that all architectures plateaued at around 13% validation accuracy in the initial transfer-learning experiments. For a 3-class problem, random guessing would give about 33% accuracy, so 13% is below chance. This surprising result is not explained. Was this a misreported metric, a consequence of severe class imbalance, or a training procedure issue? Without clarification, the motivating narrative of 'severe overfitting' is not credible.","section":"Section 2.3 and Figure 3"}],"minor_comments":[{"comment":"Typo: 'Pacreatic ductal adenocarcinoma' should be 'Pancreatic ductal adenocarcinoma'; also 'an demonstrates' should be 'and demonstrates'.","section":"Abstract"},{"comment":"The phrase 'ROC AUC of 0.989 for both classes' is imprecise; ROC AUC is a per-class or one-vs-rest measure. Clarify whether this is macro-average or a single one-vs-rest result.","section":"Section 3.4.2"},{"comment":"No confidence intervals are reported for sensitivity, specificity, or AUC. Given the small sample (239 images, 40 patients), interval estimates are essential for assessing the precision of the claimed performance.","section":"General"},{"comment":"The multi-class results in Table 2 are reported for the 'best fold' (accuracy 0.947). Selecting the best fold for headline reporting is optimistic; report mean and standard deviation across the five folds.","section":"Section 2.6 and Table 2"},{"comment":"The paper's title and framing emphasize 'early detection' of PDAC, but the experiments classify already-acquired pathology images into tissue categories, not detection in asymptomatic patients. Please temper the wording to match the actual task.","section":"Title and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The primary concern is the absence of patient-level stratification in the cross-validation. If the authors cannot demonstrate patient-independent splits, the reported AUC of 0.989 is almost certainly inflated and the paper should not be accepted. Given the clear fix (re-running with grouped CV) and the other correctable inconsistencies, I recommend major revision rather than outright rejection, but with the expectation that the revision include new results and a rigorous statistical treatment. If the re-analysis does not preserve the claimed performance, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know first: the 96.4% sensitivity/specificity and 0.989 AUC are not credible as stated, mostly because the 5-fold split is not described as patient-grouped and each patient contributes multiple images. With 40 patients and 239 images, random image-level splits would put most patients in both training and validation, inflating results. On top of that, the paper's own confusion matrix says 24/26 cancer cases (92.3%) and 27/28 non-cancer cases (96.4%), which does not match the reported 0.964 sensitivity/specificity. The abstract says frozen pretrained layers, but Section 3.4.1 describes a fully trainable network. Those are real, load-bearing inconsistencies.\n\nWhat is actually new: the dual-modality SHG/autofluorescence dataset from 40 PDAC patients is not something I have seen before, and the authors report the full arc of their experiments—six architectures, the 13% validation plateau, overfitting, class weighting, and the eventual binary model. The limitations section is honest about needing external validation. That is more transparency than most papers this size.\n\nThe main soft spot is patient leakage. The paper never says the folds are grouped by patient, and with eight ductal structures from each category per patient, the default assumption should be that they are not. That alone undermines the headline claim. The 13% initial validation accuracy is also odd—below chance for three classes—and suggests a bug in that early pipeline that was never explained. There are no confidence intervals either.\n\nFor all that, I would not desk reject. The dataset is new, the code is available, and a reviewer can ask the authors to rerun with patient-grouped folds. If the metric inconsistencies are cleaned up and the leakage question is resolved, the paper would be a useful proof-of-concept for endoscopic applications. As written, the clinical-deployment claims are unjustified.\n\nMy recommendation: send to peer review, but expect major revision. The central claim needs to be re-estimated or explicitly caveated. The paper is for people working on SHG/autofluorescence imaging of pancreatic tissue, not for a general CV audience.","headline":"The headline metrics are probably inflated by image-level cross-validation leakage and conflict with the paper's own confusion matrix, but the new dual-modality dataset and transparent reporting make it worth a referee.","tokens_in":740,"tokens_out":1736,"would_cite":false,"duration_ms":42126,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dual-modality deep learning model distinguishes pancreatic cancer from non-cancer tissue with 96.4% sensitivity and specificity.","keywords":["pancreatic ductal adenocarcinoma","PDAC early detection","deep learning classification","second harmonic generation imaging","autofluorescence imaging","ResNet-18","dual-modality medical imaging","Grad-CAM interpretability"],"falsifier":"Regroup the 5-fold cross-validation by patient so each patient's images appear in only one fold, then retrain; if sensitivity, specificity, and AUC drop materially below 0.964, 0.964, and 0.989, the original numbers were inflated by patient leakage. An external cohort test would further settle true generalization.","tokens_in":10102,"feed_emoji":"🔬","tokens_out":9232,"duration_ms":88411,"temperature":0.7,"pith_summary":"This paper tries to show that a deep learning model can catch pancreatic ductal adenocarcinoma early from optical biopsy images by combining two microscopy signals: autofluorescence and second harmonic generation (SHG). The authors analyze 239 images from 40 patients, compare six architectures, and find that a modified ResNet-18 with careful training outperforms deeper and transformer-based models on this small dataset. Their final binary classifier separates cancer from non-cancerous tissue with 96.4% sensitivity and specificity and an ROC AUC of 0.989 in 5-fold cross-validation. If true, this provides a path toward automated, interpretable PDAC detection that could supplement pathologist review and eventually pair with endoscopic optical imaging.","feed_headline":"Dual-imaging model catches pancreatic cancer at 96.4% sensitivity","feed_subtitle":"Combines autofluorescence and SHG signals to separate cancer from non-cancer tissue with an AUC of 0.989.","key_machinery":"The load-bearing object is a dual-channel image representation: autofluorescence is placed in the blue channel and the SHG signal in the green channel (scaled by 1.3 relative to autofluorescence), normalized to [0,255], then augmented with rotations up to ±30 degrees. This representation is fed into a modified ResNet-18: earlier experiments froze pretrained backbones to fight overfitting, and the final binary model is fully trainable with dropout 0.1, a 512-unit hidden layer, mixup, label smoothing, weight decay, gradient clipping, and class-weighted loss. Five-fold cross-validation generates the reported performance estimates. The mechanism lets the network combine metabolic fluorescence cu","core_discovery":"The paper reports a binary ResNet-18 classifier that takes paired autofluorescence and second-harmonic-generation (SHG) images of pancreatic tissue and labels each image as cancer or non-cancer. Trained on 239 images from 40 patients with class weighting, mixup, label smoothing, and a fully trainable head (after earlier frozen-backbone experiments), the model reaches sensitivity 0.964, specificity 0.964, ROC AUC 0.989 for both classes, precision 0.960 for cancer, and an F1 score of 0.941 in 5-fold cross-validation. The confusion matrix shows 24 of 26 cancer images and 27 of 28 non-cancer images classified correctly. The authors present this as near-perfect separation, supported by Grad-CAM v","pith_inferences":["The paper does not report patient-level grouping in the 5-fold split; if images from the same patient appear in both training and validation, the 0.964/0.989 figures may drop. A patient-level split is the natural first test.","Ablation studies that run the same pipeline on autofluorescence-only and SHG-only images would reveal which modality carries the discriminative signal; the paper does not isolate their contributions.","Pairing this classifier with existing fiber-optic SHG and autofluorescence endoscopes is a plausible next step, but in-vivo motion and lower signal quality would likely require retraining on endoscopic data.","The binary design sidesteps the model's weaker normal-versus-fibrosis separation; a deployable three-way diagnostic would need more normal-tissue samples or additional features."],"forward_implications":["If the reported performance holds outside this cohort, the model could serve as an automated second reader for PDAC on optical biopsy images, reducing reliance on manual histopathology review.","The pipeline—frozen pretrained backbones, class-weighted losses, modest augmentation, and a fully trainable smaller network—gives a workable recipe for deep learning on small medical datasets where standard fine-tuning overfits.","Because precision stays above 0.9 across recall levels, the classifier can be threshold-tuned to favor sensitivity or specificity for different clinical contexts without retraining.","The Grad-CAM maps indicate the model attends to collagen structure and relevant tissue regions, which is the kind of explanation that could help pathologists trust and verify automated findings.","The same dual-modality framework is positioned to be extended to other cancer types and to prospective or endoscopic imaging settings."],"supporting_citations":[{"why":"Establishes PDAC's low survival and late detection, the clinical problem the framework targets.","marker":"1"},{"why":"Provides prior SHG collagen anisotropy analysis on human pancreatic cancer, grounding the SHG channel's biological signal.","marker":"18"},{"why":"Shows SHG microscopy detects extracellular-matrix and fibrosis changes, justifying the dual-modality choice.","marker":"17"},{"why":"Links aligned stromal collagen to PDAC prognosis, giving a pathological rationale for using collagen patterns to separate cancer from non-cancer.","marker":"21"},{"why":"Supplies the ResNet architecture that the final ResNet-18 classifier is built on.","marker":"27"},{"why":"Defines the Vision Transformer baselines whose overfitting on the small dataset motivates the frozen-backbone approach and ResNet choice.","marker":"28"},{"why":"Provides the k-fold cross-validation methodology used for the final performance estimates.","marker":"29"},{"why":"Demonstrates fiber-optic SHG endomicroscopy feasibility, supporting the claim that the classifier could combine with endoscopic imaging.","marker":"24"}],"fun_headline_variants":["Dual-imaging AI catches pancreatic cancer at 96.4% sensitivity","Autofluorescence + SHG imaging AI detects pancreatic cancer with 96.4% sensitivity","ResNet-18 on dual-modality images achieves AUC 0.989 for pancreatic cancer","Dual-imaging AI model hits AUC 0.989 for pancreatic cancer"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported accuracy assumes the cross-validation split keeps all images from the same patient in the same fold; the paper never states this, so the near-perfect scores could partly come from patient-specific features rather than general cancer patterns.","fun_headline_variants_meta":{"raw":{"variants":["Dual-imaging AI catches pancreatic cancer at 96.4% sensitivity","Autofluorescence + SHG imaging AI detects pancreatic cancer with 96.4% sensitivity","ResNet-18 on dual-modality images achieves AUC 0.989 for pancreatic cancer","Dual-imaging AI model hits AUC 0.989 for pancreatic cancer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001223,"raw_usage":{"total_tokens":4882,"prompt_tokens":779,"completion_tokens":4103,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":4015}},"tokens_in":523,"tokens_out":4103,"duration_ms":26516,"temperature":1.0,"reasoning_tokens":4015,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:43:29.527682+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regroup the 5-fold cross-validation by patient so each patient's images appear in only one fold, then retrain; if sensitivity, specificity, and AUC drop materially below 0.964, 0.964, and 0.989, the original numbers were inflated by patient leakage. An external cohort test would further settle true generalization.","supporting_citations":[{"cited_title":"G.; Karamouzis, M","cited_arxiv_id":null,"evidence_quote":"Establishes PDAC's low survival and late detection, the clinical problem the framework targets."}],"review_version":1}