{"id":"d89472ab-6e29-4c6a-846d-3d85c4c11782","arxiv_id":"2411.15274","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"VERN, a graph-based multiple instance learning model, predicts spread through air spaces from lung cancer whole-slide images with internal AUC 0.92 and external AUC 0.70 to 0.88.","lead":"A deep learning model called VERN was built to predict a lung cancer invasion pattern called STAS from whole-slide pathology images, reporting high accuracy in-house and lower but still meaningful accuracy on outside datasets. The paper is worth reading because STAS status can influence whether a patient receives a limited or a larger lung resection, so an automated slide-level test could directly affect surgical decisions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The internal cross-validation may split slides from the same patient across train and test, which could inflate the reported AUCs; the paper does not state that patient-level grouping was used.","rationale":"The reader identified the same load-bearing concern: the internal split may not be patient-independent, and external slide labels may be unreliable. I agree with the reader's conditional verdict. The patient-independence issue is the most consequential because it bears directly on the internal AUCs that support the 'clinical-grade' framing, and it is not a matter of disagreement with consensus but a potential internal inconsistency in the evaluation protocol. The paper's external validation on TCGA and CPTAC, plus the released code and features, suggests a real signal, so a rejection is not warranted; however, the headline claim is not secure until the split is corrected or explicitly shown to be patient-level. My proposed concrete test would settle the issue with modest effort. The verdict should remain as the reader gave it: CONDITIONAL or, equivalently, no change to the reader's conditional recommendation.","tokens_in":18185,"tokens_out":2119,"duration_ms":22390,"concrete_test":"Using the released code and extracted features, regroup the 1,190 internal slides by patient ID so that all FS and PS slides from one patient are assigned to the same fold, then rerun the five-fold cross-validation and the internal test. Compare the patient-level mean AUC, the standard deviation across folds, and the best-fold AUC against the reported 0.8683 and 0.9215. If the patient-level AUC drops by more than about 0.03 or the best-fold AUC falls below the reported external validation results, the clinical-grade claim is not supported. Also, plot the fold-wise AUCs and report 95% confidence intervals to check whether the best-fold number is an outlier.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that VERN achieves clinical-grade STAS prediction depends on unbiased internal evaluation. Each of the 356 patients contributed multiple slides (one FS and several PSs), yielding 1,546 slides, of which 1,190 were used for internal validation and testing. The methods state that five-fold cross-validation divided the internal validation set into five subsets, but they never state that all slides from a single patient were kept in the same fold. If the split is slide-level, the same patient's FS and PSs can appear in both training and validation, so the model could learn patient-specific artifacts (staining, scanning, or tissue-preparation signatures) rather than generalizable STAS morphology. This would directly inflate the average AUC of 0.8683 and the headline best-fold AUC of 0.9215, and it would also bias the comparison against the MIL baselines, whose graph-based counterpart could benefit similarly. The paper also reports the best fold rather than the full fold distribution, and the external validation AUCs (0.7029-0.9181) are lower, which is consistent with some internal optimism. The most load-bearing assumption is therefore that the internal splits are patient-independent, and the manuscript provides no evidence for that assumption. A second concern, that two pathologists labeled TCGA/CPTAC slides without reporting inter-observer agreement, is also relevant but secondary, because the internal leakage alone would already undermine the headline performance claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VERN, a feature-interactive Siamese graph encoder for predicting spread through air spaces (STAS) from lung cancer whole-slide images. The authors constructed a single-cohort dataset of 1,546 frozen and paraffin-embedded sections from 356 patients, trained VERN with five-fold cross-validation, and report an internal AUC of 0.9215 (best fold; average 0.8683), plus AUCs of 0.8275 and 0.8829 on frozen and paraffin test sections. They also validate on 356 single-cohort slides and on external slides from Zhengzhou/Henan Cancer Hospital, TCGA, and CPTAC, and provide an open web platform. The central claim is that VERN achieves clinical-grade STAS prediction and outperforms existing multiple-instance learning baselines owing to its spatial graph construction.","tokens_in":18409,"tokens_out":3006,"duration_ms":31864,"significance":"If the reported results hold, this would be one of the first WSI-level STAS predictors and a potentially useful clinical decision-support tool, especially given the low sensitivity of frozen-section STAS assessment by pathologists. The work has clear strengths: it compares against five MIL baselines, provides interpretability heatmaps, publishes code and an online platform, and includes three external datasets. However, the significance is conditional on the internal evaluation being unbiased; the manuscript currently does not establish patient-level independence in the cross-validation split, and the headline number is the best of five folds rather than a central estimate.","major_comments":[{"comment":"The internal cross-validation split is described only at the slide level: 'the internal validation set, which includes both FSs and PSs, into five subsets.' The manuscript never states that all slides from a single patient were kept in the same fold. Since 1,546 slides come from 356 patients, with each patient contributing one FS and several PSs, a slide-level split can place slides from the same patient in both training and validation. This would let the model exploit patient-specific staining, scanning, or tissue-preparation signatures rather than learning generalizable STAS morphology, directly inflating the reported average AUC of 0.8683 and the best-fold AUC of 0.9215, and also biasing the comparison against MIL baselines. Please specify whether patient-level grouping was used, or re-run the cross-validation with patient-stratified folds and report the resulting performance.","section":"Methods: Clinical single cohort and multicenter data collection; Results (first paragraph)"},{"comment":"The abstract and Results highlight an AUC of 0.9215, but this is the best of five cross-validation folds; the average in-domain AUC reported in Table 1 and Figure 2c is 0.8683. Selecting the best fold as the headline performance overstates the expected prospective accuracy. Please report all five fold-level AUCs with confidence intervals, and state explicitly whether any model selection was performed across folds. If the best fold was chosen post hoc, the reported performance is optimistic.","section":"Results, Figure 2 and Table 1"},{"comment":"The 356-slide 'single-cohort validation' is not an independent patient-level validation set. The Methods state that these are additional PSs from the same single cohort and that they 'had not been internally trained, verified, and tested,' but the same 356 patients contributed FSs and PSs used in internal training. If any slide from a patient was in the training set, then the patient's STAS status and slide characteristics are known to the model during training, so the AUC of 0.9181 in Figure 4a reflects within-patient correlation and cannot be cited as evidence of generalizability. Please either restrict this validation to patients entirely excluded from training or reframe it as a within-cohort slide-type test, and rely on the truly external datasets for generalization claims.","section":"Results: Single-cohort and multicenter STAS validation sets; Methods: Clinical single cohort and multicenter data…"},{"comment":"The external TCGA and CPTAC labels were assigned by two pathologists, but the manuscript reports no inter-observer agreement (e.g., Cohen's kappa), no description of the labeling protocol's blinding, and no verification against a reference standard. Given that STAS diagnosis is known to be subjective and that even intraoperative frozen-section assessment has moderate agreement, the reliability of these external labels is load-bearing for the external AUCs (0.7029 and 0.7555). Please report the labeling procedure in detail, including the criteria used, whether disagreements were adjudicated, and the inter-observer agreement.","section":"Methods: Clinical single cohort and multicenter data collection"}],"minor_comments":[{"comment":"The caption reads 'with the diagonal line showing the in-domain test results'; in an ROC plot the diagonal line is chance-level performance, not the test results. Please clarify what the diagonal line represents.","section":"Figure 2a caption"},{"comment":"The terms 'histopathological images,' 'slides,' and 'sections' are used interchangeably; please standardize terminology (e.g., 'whole-slide images' vs. 'patches') to avoid confusion about the evaluation unit.","section":"Abstract and Results"},{"comment":"The grid search is described only as tuning 'learning rate, regularization strength, and batch size,' but only the learning rate and optimizer settings are reported. Please list the ranges considered and the final hyperparameter values for reproducibility.","section":"Methods: Experimental setup and implementation details"},{"comment":"The table header contains 'PRC' in the caption text, but the table reports AUC and other metrics. Please ensure metric abbreviations are defined consistently (e.g., use AUROC and AUPRC).","section":"Results, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central technical novelty (spatial graph encoding for STAS) is plausible and the external datasets provide some independent support. However, the missing patient-level split and the reporting of the best fold rather than the average are load-bearing issues that prevent acceptance in the current form. If the authors can re-run the internal evaluation with patient-stratified folds and report all fold results, the paper may become suitable. I did not independently verify the availability or functionality of the GitHub repository or the online prediction platform."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on the VERN paper. The genuinely new thing is the application: WSI-level STAS prediction with a graph encoder, and as far as the cited literature goes, it's the first for that endpoint. The paper also ships code, extracted features, and a public demo site, which is real evidence and makes the work checkable. The comparison against ABMIL, DSMIL, TransMIL, DTFD-MIL, and IBMIL is standard and fair; VERN's average internal AUC of 0.8683 beats the best baseline by about 0.09. External validation on two hospital cohorts plus TCGA and CPTAC is more than most papers in this space do.\n\nThe soft spots are the three the reader's note catches, and I think they're ordered correctly. The load-bearing one is that the internal five-fold split is never stated to be patient-independent. Each of the 356 patients contributed multiple slides—one FS plus several PSs—and the methods only describe slide-level division (1190 slides into internal validation and test, then five folds). If the same patient's slides appear in training and validation, the model can learn patient-specific artifacts, and the reported AUCs are inflated. That concern is not hypothetical; it's the difference between a clinical tool and a staining artifact detector. The second issue is reporting: the headline 0.9215 comes from the best fold, while the mean is 0.8683, and no fold distribution is given. That's an easy fix but it should be done. Third, the TCGA/CPTAC labels are from two pathologists with no inter-observer agreement, which matters for a subjective endpoint. The external AUCs (0.70–0.76 on TCGA/CPTAC) are lower than the internal numbers, consistent with some optimism, but still above chance and worth explaining.\n\nNone of these issues is disqualifying. The method is coherent, the baselines are appropriate, and the artifacts make the split question independently verifiable. The 'clinical-grade' framing is over-claimed, but that's a wording issue. The right fix is patient-level splitting, fold-range reporting, and a tempering of the conclusion.\n\nWho should read this: computational pathologists and thoracic oncology researchers. It deserves a serious referee—the endpoint is clinically relevant and the external validation is a strong asset. I'd send it to review with a request for the re-analysis, not desk-reject.","headline":"A solid first WSI-level STAS prediction model with genuine external validation, but the internal split needs to be patient-level before the headline AUC is taken at face value.","tokens_in":19006,"tokens_out":3655,"would_cite":false,"duration_ms":34821,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A feature-interactive Siamese graph encoder, VERN, predicts spread through air spaces (STAS) in lung cancer whole-slide images, reporting an internal AUC of 0.9215 and frozen- and paraffin-section test AUCs of 0.8275 and 0.8829.","keywords":["spread through air spaces","lung cancer","whole slide image","graph neural network","Siamese network","histopathology","multiple instance learning","computer-aided diagnosis"],"falsifier":"Re-run the five-fold cross-validation with all slides grouped by patient before splitting, so no patient appears in both training and test folds, and compare the resulting AUC with the reported 0.9215; a large drop would show the model was exploiting patient-specific slide artifacts rather than STAS biology.","tokens_in":17940,"feed_emoji":"🔬","tokens_out":9047,"duration_ms":83670,"temperature":0.7,"pith_summary":"Spread through air spaces (STAS) is an aggressive invasion pattern in lung cancer: tumor cells drift through air spaces beyond the main tumor, and its presence can change whether a surgeon removes only the nodule or a whole lobe. The paper proposes VERN, a feature-interactive Siamese graph encoder that predicts STAS directly from whole-slide histopathology images by turning each slide into a spatial graph of tissue patches. In their experiments VERN reached an AUC of 0.9215 in internal validation, 0.8275 on frozen test sections and 0.8829 on paraffin test sections, and held up across external validation cohorts. If these numbers hold, VERN would give pathologists and surgeons an automated second reader for a decision that currently hinges on subjective, time-consuming slide review.","feed_headline":"Graph AI predicts tumor spread in lung cancer slides","feed_subtitle":"Whole-slide heatmaps flag tumor edges where spread-through-air-spaces cells hide, aiding surgical decisions.","key_machinery":"The load-bearing object is VERN itself, a feature-interactive Siamese graph encoder. Each whole-slide image is converted into a spatial topological graph: patches become nodes, patch features come from two pretrained extractors (1024- and 768-dimensional), and a K-nearest-neighbor rule (K=9) draws edges between nearby patches, so the graph encodes where tissue structures sit relative to one another. VERN then runs two symmetric encoder branches, each a sequence of GCNConv, SAGEConv, ReLU, Dropout, MLP and Rescale layers; the branches share weights and exchange information through cross-graph message passing, and a skip connection carries the original input forward. The two branch outputs are concatenated and averaged for the slide-level STAS prediction, and per-patch attention contributions are normalized and mapped back onto the slide as heatmaps.","core_discovery":"On the paper's own terms, the discovery is that STAS—a pattern in which tumor cells spread through alveolar air spaces beyond the main tumor—can be predicted from whole-slide histopathology images at the slide level, and that representing the slide as a spatial graph is what makes this work. The authors report an internal validation AUC of 0.9215, frozen-section and paraffin-section test AUCs of 0.8275 and 0.8829, and external cohort AUCs in the range of roughly 0.70 to 0.92. They also report that VERN outperforms five multiple instance learning baselines under the same five-fold cross-validation protocol. The paper frames this as the first whole-slide-image-level STAS prediction method and as a possible intraoperative aid, since frozen-section STAS is clinically important but hard for pathologists to detect.","pith_inferences":["The paper does not report an ablation that replaces the K-nearest-neighbor spatial graph with random edges; such an ablation would directly test whether spatial topology, rather than the extra parameters alone, drives the improvement over multiple instance learning baselines.","Because each patient contributed multiple slides, a patient-level fusion rule (any slide positive implies STAS) and a patient-level AUC would make the clinical utility figures more interpretable; the paper mentions the rule but does not evaluate it.","The same architecture could be transferred to other histopathology tasks where spread or adjacency matters, such as lymphovascular invasion or perineural invasion, provided a similar frozen-plus-paraffin training set is available.","An external reader study comparing VERN against multiple pathologists on the same slides would place the reported AUCs in a clinical context; the paper compares with literature numbers rather than a head-to-head reader study."],"forward_implications":["If VERN's accuracy holds, intraoperative frozen-section review gains a rapid second reader that could cut missed STAS cases before the surgeon chooses resection extent.","The per-patch attention maps give pathologists a concrete region to re-check, namely the tumor edge and peritumoral air spaces, rather than scanning the entire slide.","The graph representation is the claimed reason for the gains, so future whole-slide classifiers for other spatially distributed patterns could adopt the same encoder.","Because the model was trained on both frozen and paraffin sections, its frozen-section predictions benefit from paraffin-derived prior knowledge, a training strategy that could be reused elsewhere."],"supporting_citations":[{"why":"the ABMIL baseline that VERN is compared against in Table 1.","marker":"[32]"},{"why":"the DSMIL baseline used in the comparison.","marker":"[33]"},{"why":"the TransMIL baseline used in the comparison.","marker":"[34]"},{"why":"the DTFD-MIL baseline used in the comparison.","marker":"[35]"},{"why":"the IBMIL baseline used in the comparison.","marker":"[36]"},{"why":"supplies the 1024-dimensional patch feature extractor used to build node features.","marker":"[57]"},{"why":"supplies the 768-dimensional patch feature extractor used to build node features.","marker":"[58]"},{"why":"the K-nearest-neighbor algorithm used to draw edges of the spatial topological graph.","marker":"[61]"},{"why":"provides the GCNConv layer used inside each branch of the Siamese encoder.","marker":"[63]"},{"why":"provides the SAGEConv layer used inside each branch of the Siamese encoder.","marker":"[64]"}],"fun_headline_variants":["Graph AI spots STAS on lung cancer whole slides","Slide-level graph AI predicts STAS in lung cancer","Graph-based AI predicts STAS from lung pathology images","Whole-slide graph AI predicts lung cancer STAS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that all slides from the same patient stayed in the same training or test split—if a patient's frozen and paraffin slides were allowed to straddle the split, the model could be recognizing patient-specific slide artifacts rather than STAS itself, which would inflate the reported AUCs.","fun_headline_variants_meta":{"raw":{"variants":["Graph AI spots STAS on lung cancer whole slides","Slide-level graph AI predicts STAS in lung cancer","Graph-based AI predicts STAS from lung pathology images","Whole-slide graph AI predicts lung cancer STAS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001353,"raw_usage":{"total_tokens":5480,"prompt_tokens":916,"completion_tokens":4564,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":4501}},"tokens_in":532,"tokens_out":4564,"duration_ms":28897,"temperature":1.0,"reasoning_tokens":4501,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:40:04.345464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the five-fold cross-validation with all slides grouped by patient before splitting, so no patient appears in both training and test folds, and compare the resulting AUC with the reported 0.9215; a large drop would show the model was exploiting patient-specific slide artifacts rather than STAS biology.","supporting_citations":[{"cited_title":"Attention -based deep multiple instance learning,","cited_arxiv_id":null,"evidence_quote":"the ABMIL baseline that VERN is compared against in Table 1."},{"cited_title":"Dual-stream multiple instance learning network for whole slide image classification with self- supervised contrastive learning,","cited_arxiv_id":null,"evidence_quote":"the DSMIL baseline used in the comparison."},{"cited_title":"Transmil: Transformer based correlated multiple instance learning for whole slide image classification,","cited_arxiv_id":null,"evidence_quote":"the TransMIL baseline used in the comparison."},{"cited_title":"A fuzzy k-nearest neighbor algorithm,","cited_arxiv_id":null,"evidence_quote":"the K-nearest-neighbor algorithm used to draw edges of the spatial topological graph."},{"cited_title":"Inductive Representation Learning on Large Graphs,","cited_arxiv_id":null,"evidence_quote":"provides the SAGEConv layer used inside each branch of the Siamese encoder."}],"review_version":1}