{"id":"2c900c18-7000-45da-b85b-065b1c7548b8","arxiv_id":"2505.00134","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Prompt engineering and anatomical context significantly affect zero-shot diagnostic accuracy of pathology vision-language models, with CONCH outperforming a larger model, Quilt-LLAVA.","lead":"This paper tests how three vision-language models (Quilt-Net, Quilt-LLAVA, CONCH) perform at zero-shot cancer diagnosis on 3,507 digestive whole slide images under different prompt phrasings. It finds that CONCH is most accurate with precise anatomical prompts, while prompt wording and anatomical detail strongly affect all models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline ordering rests on per-patch ROC/AUC with slide labels inherited per patch; no slide-level pooling is described, so the AUC comparisons are uninterpretable as diagnostic accuracy.","rationale":"The reader identified the missing patch-to-slide aggregation as the weakest assumption; I agree that this is the most load-bearing concern. The central claim of the paper is that prompt engineering, specifically anatomical precision, significantly impacts diagnostic performance, with CONCH best under precise anatomical prompts. All quantitative support for this claim comes from ROC/AUC values in Section 5.1. If those values are computed per patch with slide-level labels inherited by every patch, they do not measure slide-level diagnostic accuracy, and the model comparisons and prompt-orderings are not interpretable. The absence of any description of how patch scores were pooled, or how Quilt-LLAVA's binary outputs entered a ROC analysis, makes the main result unverifiable as reported. A re-analysis with explicit slide-level pooling could rescue the claim, so the appropriate verdict remains CONDITIONAL rather than REJECT. I also note secondary concerns such as prompt selection on the same test set and lack of error bars, but the aggregation issue is the one that most directly undermines the central claim.","tokens_in":10563,"tokens_out":3856,"duration_ms":43139,"concrete_test":"Recompute the results of Figures 3 and 4a using explicit slide-level pooling: for each WSI, aggregate the patch scores under at least three rules (mean, maximum, and 95th percentile) into one slide-level score, then construct ROC curves from the 3,507 slide-level scores. Check whether the P5 > P6 > P7 > P8 degradation persists for Quilt-Net, Quilt-LLAVA, and CONCH under all three pooling rules. Also report exactly how Quilt-LLAVA's binary patch labels were converted to a ROC score. If the ordering changes or flips under any reasonable pooling rule, the central claim about anatomical precision is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is that the central quantitative claim is evaluated at patch level while the paper speaks to diagnostic accuracy at WSI level. In Section 4, WSIs are tiled into 512x512 patches at 5x with 0% overlap; in Section 5.1, ROC curves and AUCs are reported but no patch-to-slide pooling is specified. Because each WSI carries one slide-level invasiveness label, every patch from an invasive slide is implicitly labelled invasive even if it depicts normal mucosa, stroma, or background. With thousands of patches per slide, patch-level AUC is dominated by how many patches a model flags and by slide size, not by correct slide diagnosis; it also massively inflates the effective sample size. The problem is clearest for Quilt-LLAVA: Section 5.2 states it does not produce continuous scores and that binary labels are used for heatmaps, yet Section 5.1 reports ROC curves for it without explaining how a ROC can be formed from binary patch labels. Therefore the reported ordering P5 > P6 > P7 > P8 and the claim that anatomical precision drives performance may be an artifact of the unspecified unit of analysis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports a systematic zero-shot evaluation of three vision-language models (Quilt-Net, Quilt-LLAVA, CONCH) on a large in-house digestive pathology dataset of 3,507 WSIs. The authors design nine invasiveness prompts and three dysplasia prompts that vary detail specificity, anatomical precision, instructional framing, and output constraints; they report ROC/AUC for invasiveness and dysplasia, per-tissue AUC values, a magnification-level analysis, and pathologist-reviewed attention heatmaps. The central claims are that prompt engineering significantly changes zero-shot diagnostic performance, that CONCH is the best model when given precise anatomical references, and that reducing anatomical precision consistently degrades performance.","tokens_in":10752,"tokens_out":5688,"duration_ms":53930,"significance":"The study has several strengths: it evaluates three publicly available pathology VLMs, uses a clinically realistic large dataset spanning seven digestive tissue types, provides a structured ablative prompt framework, and includes qualitative attention-map validation by a board-certified pathologist. If the evaluation protocol were fully specified and statistically grounded, the paper would offer practically useful evidence on prompt sensitivity in zero-shot pathology VLMs and on the limited role of model scale relative to domain alignment. However, the central quantitative claims are not currently verifiable because the unit of analysis is ambiguous, and the paper does not report uncertainty estimates. The significance of the reported model and prompt ordering therefore cannot yet be assessed.","major_comments":[{"comment":"The manuscript does not specify how patch-level outputs are pooled into the slide-level ROC/AUC curves. Section 4 states that WSIs are tiled into 512×512 patches at 5× with a sliding window and 0% overlap, while Section 5.1 reports ROC curves for invasiveness; each WSI has a single slide-level label (Table 1). If every patch inherits the slide label, then non-invasive tissue within an invasive WSI (e.g., normal mucosa, stroma, background) is treated as invasive, and the reported AUC reflects patch-count weighting and slide size rather than slide-level diagnostic accuracy. This is load-bearing for the headline ordering P5 > P6 > P7 > P8 and the average AUC ranking (CONCH 0.876 > Quilt-LLAVA 0.753 > Quilt-Net 0.666). Please specify the exact aggregation rule (e.g., mean or max patch score per slide, or a multiple-instance model) and report slide-level ROC/AUC with the corresponding effective sample size.","section":"§4 and §5.1 (Figs. 3–4)"},{"comment":"There is a direct contradiction between the attention-map protocol and the ROC analysis for Quilt-LLAVA. Section 5.2 states that Quilt-LLAVA 'does not inherently produce continuous confidence scores; therefore, binary labels are used' for heatmaps, yet Section 5.1 and Figure 3(b) report ROC curves and an AUC of 0.807 for Quilt-LLAVA under P5. A ROC curve requires a continuous score or at least a ranking; with binary patch labels the ROC degenerates to a single operating point and the AUC becomes a fraction of positive pairs, not a threshold-averaged measure. Please clarify how the ROC was constructed for Quilt-LLAVA, or restrict its evaluation to the appropriate metric.","section":"§5.2 vs. §5.1"},{"comment":"No error bars, confidence intervals, or significance tests are reported anywhere in the results. The dataset contains 3,507 slides, but if the analysis is per-patch the effective sample size is inflated by thousands of patches per slide; if per-slide pooling is used, the variance across slides should be reported. The per-tissue AUCs in Table 3 are especially affected, as classes such as AW (n=108), MCW (n=45), and GJ (n=40) are small, yet differences such as GJ=0.94 vs AW=0.59 are interpreted without uncertainty. The differences used to support claims such as the degrading effect of anatomical precision and the average AUC ordering may be within sampling variability. Please provide bootstrap or other uncertainty estimates at the slide level and, where appropriate, statistical tests for the prompt and model comparisons.","section":"§5.1, Table 3, Figs. 3–4"}],"minor_comments":[{"comment":"There is a typo: 'Quitl-LLAVA' should be 'Quilt-LLAVA' in the sentence reporting the P5 AUC comparison.","section":"§5.1"},{"comment":"The title mentions 'Efficient Prompt Design,' but the paper reports no efficiency metrics such as inference time, GPU hours, or token counts; either add such measurements or adjust the title and framing.","section":"Title and §2"},{"comment":"The magnification-level analysis does not state which prompt template is used, nor does it provide confidence intervals or the number of slides per magnification level; please specify these details.","section":"Fig. 6"},{"comment":"The prompt framework is described as 'based on information theory and clinical communication principles,' but no formal derivation or supporting reference is provided; a brief justification or citation would strengthen the presentation.","section":"§4"},{"comment":"The paper does not mention whether multiple WSIs can originate from the same patient; if they can, slide-level clustering should be accounted for in the statistical analysis.","section":"§3 and §5.1"},{"comment":"The prompt templates and the best-prompt conclusions were developed and evaluated on the same dataset; a validation split or an explicit acknowledgment of this limitation would help temper the generalizability claims.","section":"§4 and §5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and addresses a timely question in computational pathology. My recommendation of major revision is driven by the unspecified patch-to-slide aggregation and the Quilt-LLAVA ROC inconsistency, both of which are fixable with additional analysis. The attention-map validation by a pathologist is a valuable qualitative complement. I do not see grounds for rejection, but the authors should provide slide-level results, uncertainty quantification, and a clearer statistical protocol before the central claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a legitimate empirical probe of prompt sensitivity in pathology VLMs, but the headline AUC numbers are built on an unspecified patch-to-slide aggregation, and for Quilt-LLAVA the reported ROC curve is either unexplained or nonsensical. I would still send it to peer review, because the question is worth answering and the flaws are fixable.\n\nWhat is new: the authors systematically vary prompt dimensions for three VLMs on a 3,507-slide digestive pathology dataset (private), across seven tissue types, and report zero-shot performance. That is a real ablation, and the observation that anatomical precision affects performance aligns with prior work but is quantified here in a clinically relevant setting. The attention-map analysis with a pathologist's qualitative review on four WSIs is also genuinely useful, even if illustrative.\n\nCredit where due: the prompt framework is clearly described, the tissue-by-tissue breakdown is informative, and the comparison between CONCH and Quilt-LLAVA supports the claim that domain alignment can matter more than raw parameter count. Those are solid empirical observations.\n\nSoft spots: the load-bearing issue is that ROC/AUC is reported without stating how patch predictions become slide-level scores. Each WSI carries one label; if every patch inherits that label, a normal patch from an invasive slide is called invasive and the effective sample size is inflated by thousands of patches per slide. That makes the reported AUCs unverifiable as diagnostic accuracy. The problem is worse for Quilt-LLAVA: Section 5.2 says it does not produce continuous scores and binary labels are used for heatmaps, yet Section 5.1 plots a ROC curve for it with no explanation of how a threshold-varying curve is derived from binary outputs. No error bars or significance tests appear anywhere, and the best prompt is selected on the same test set used to report performance, so the 'P5 is optimal' claim is overconfident. The concordance study is only four slides; fine as a demonstration, not as validation.\n\nThese are addressable. Add a clear pooling rule (e.g., slide score = max or 95th percentile of patch scores), report bootstrap confidence intervals over slides, hold out a validation set for prompt selection, and either drop the Quilt-LLAVA ROC or explain the score used. If they do that, this becomes a modest but solid contribution.\n\nWho it is for: anyone building zero-shot pathology pipelines who wants a starting point for prompt design and evidence on how model choice interacts with phrasing. A serious referee should engage; this is not a desk reject.","headline":"Useful probe of prompt sensitivity in pathology VLMs, but the AUC numbers rest on an unspecified patch-to-slide pooling and one model that may not support ROC at all.","tokens_in":11294,"tokens_out":2322,"would_cite":false,"duration_ms":24747,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt wording—especially naming the exact organ—decides how well vision-language models diagnose cancer from whole-slide images, and the domain-trained CONCH beats a far larger model when the prompt gives precise anatomical context.","keywords":["vision-language models","computational pathology","prompt engineering","zero-shot classification","histopathology","whole slide images","anatomical precision","cancer invasiveness"],"falsifier":"Recompute slide-level AUCs after defining an explicit patch-to-slide pooling rule (e.g., mean or max of patch probabilities per slide) and re-rank the three models; if the CONCH-versus-Quilt-LLAVA ordering or the anatomical-precision gradient changes under a different pooling rule, the conclusion that prompt anatomy drives performance would not generalize.","tokens_in":10338,"feed_emoji":"🔬","tokens_out":6164,"duration_ms":54790,"temperature":0.7,"pith_summary":"The paper tries to establish that how a prompt is worded—particularly whether it names the exact organ and tissue—changes whether off-the-shelf vision-language models can detect invasive cancer and dysplasia in whole slide images. On an in-house set of 3,507 digestive-tract slides, the authors compare three models (Quilt-Net, Quilt-LLAVA, and CONCH) without any fine-tuning. They find that CONCH, a model trained specifically on histopathology image-text pairs, reaches the highest accuracy when the prompt gives precise anatomical context, and that all models lose accuracy when anatomical detail is removed. They also argue that a much larger model (Quilt-LLAVA, roughly 7 billion parameters) does not beat a smaller domain-trained one (CONCH, roughly 200 million), so domain alignment matters more than scale. The practical stakes are that prompt design becomes a cheap, parameter-free lever for improving AI-assisted pathology diagnosis.","feed_headline":"Prompt wording, not model size, drives pathology AI accuracy","feed_subtitle":"On 3,507 digestive slides, naming the organ in the prompt lifts CONCH's AUC to 0.935, and accuracy falls as anatomy is removed.","key_machinery":"The machinery is the prompt-engineering framework itself: a set of nine prompt templates spanning four dimensions—detail specificity (LOW/MEDIUM/HIGH), anatomical precision (LOW/MEDIUM/HIGH), instructional framing (MINIMAL/TASK/EXPERT), and output constraints (EXPLICIT/IMPLICIT)—plus the three pretrained VLMs run zero-shot. The templates P1–P9 and D1–D3 are the experimental instruments: by holding three dimensions fixed and varying one, the paper isolates what in the linguistic input moves the AUC. The load-bearing comparison is prompt P5 (high anatomical precision, medium specificity) against P6–P8 (progressively vaguer anatomy), which directly tests the anatomical-context hypothesis, and the model comparison CONCH versus Quilt-LLAVA, which tests scale-versus-alignment.","core_discovery":"The central claim is that zero-shot performance of vision-language models on pathology diagnostics is highly sensitive to prompt composition, and that the best prompt is one with intermediate information density, high anatomical precision, and explicit output constraints. In the ablative study, prompt P5—which names the organ (e.g., colon wall, lymph node) and asks for a single-word label—produced the highest AUC for all three models, with CONCH reaching 0.935 for invasive cancer classification. When the anatomical reference was weakened (prompts P6–P8, moving from organ to 'gastrointestinal system' to 'digestive system' to none), performance consistently dropped. The paper further shows that near-synonymous medical terms ('dysplasia', 'atypia', 'precancerous') shift which prompt works best for each model, and that expert-role framing ('You are an expert pathologist') hurts or is neutral in two of the three models. The conclusion is that domain-specific pretraining and careful prompt anatomy, not parameter count, determine diagnostic accuracy, and the authors position this as a starting guideline for prompt engineering in computational pathology.","pith_inferences":["A testable extension would be a broader grid over organ taxonomies (adding laterality, biopsy site, or patient age); the benefit of anatomical precision may be an inverted-U rather than monotonic, with diminishing or negative returns beyond a moderate level.","The patch-labeling shortcut (every patch inherits the slide label) is common, but re-analysis with attention-based multiple-instance pooling could narrow the CONCH-versus-Quilt-LLAVA gap, since Quilt-LLAVA's patch-level scores are binary.","The near-synonym result suggests these models rely on lexical associations from pretraining rather than reasoning about underlying pathology; a probe with adversarial synonyms (e.g., 'malignant' vs 'cancerous') could test how systematic the sensitivity is.","Clinically, the finding implies prompt design should be part of regulatory evaluation of AI pathology tools, since a favorable AUC could in part be a prompt artifact."],"forward_implications":["Zero-shot pathology screening can be improved with no retraining by naming the exact organ in the prompt; a single phrase change is worth more than swapping in a larger model.","Model rankings in medical VLM benchmarks are unstable across prompt wordings, so evaluations should report a family of prompts rather than a single template.","For clinical deployment, prompts with explicit output constraints and medium information density are a sensible default starting point.","The choice of terminology for a condition (dysplasia vs atypia vs precancerous) should be matched to the model, since the best synonym differs by model.","Attention-map behavior tracks diagnostic quality: the model with the highest AUC also highlights invasive cancer and precursor lesions in a way a pathologist recognizes."],"supporting_citations":[{"why":"Introduces Quilt-1M, the million image-text dataset that Quilt-Net is fine-tuned on and that defines the model's domain alignment.","marker":"[12]"},{"why":"Introduces CONCH, the vision-language foundation model that achieves the highest accuracy in the study.","marker":"[16]"},{"why":"Introduces Quilt-LLAVA, the 7B-parameter instruction-tuned model whose performance is compared against CONCH.","marker":"[21]"},{"why":"Describes the LLAVA visual-instruction-tuning framework that Quilt-LLAVA's architecture is based on.","marker":"[14]"},{"why":"Defines the CLIP contrastive dual-encoder paradigm that Quilt-Net's architecture adapts for histopathology.","marker":"[20]"},{"why":"Presents CoCa, the contrastive-captioner architecture underlying CONCH's decoupled decoder design.","marker":"[26]"},{"why":"Supplies the hierarchical self-supervised pretraining approach that Quilt-Net builds on for gigapixel slides.","marker":"[3]"},{"why":"Documents prompt-sensitivity of VLMs in oncology, motivating the paper's systematic prompt ablations.","marker":"[5]"}],"fun_headline_variants":["For zero-shot pathology, prompt anatomy beats model size","Naming the organ in prompts lifts pathology VLM AUC","CONCH hits 0.935 AUC when prompt names the organ","Prompt precision > model complexity in pathology AI","Zero-shot pathology: anatomy in prompts is key"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that every 512×512 patch cut from a slide carries the slide's own diagnosis, and that the way patch scores are turned into a slide-level ROC curve is valid—but it never says how that pooling is done, so a patch of normal tissue next to a tumor is counted as tumor.","fun_headline_variants_meta":{"raw":{"variants":["For zero-shot pathology, prompt anatomy beats model size","Naming the organ in prompts lifts pathology VLM AUC","CONCH hits 0.935 AUC when prompt names the organ","Prompt precision > model complexity in pathology AI","Zero-shot pathology: anatomy in prompts is key"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1504,"prompt_tokens":1035,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":651,"tokens_out":469,"duration_ms":4859,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:49:59.729633+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute slide-level AUCs after defining an explicit patch-to-slide pooling rule (e.g., mean or max of patch probabilities per slide) and re-rank the three models; if the CONCH-versus-Quilt-LLAVA ordering or the anatomical-precision gradient changes under a different pooling rule, the conclusion that prompt anatomy drives performance would not generalize.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Quilt-1M, the million image-text dataset that Quilt-Net is fine-tuned on and that defines the model's domain alignment."},{"cited_title":"Nature Medicine30(3), 863–874 (2024) Title Suppressed Due to Excessive Length 17","cited_arxiv_id":null,"evidence_quote":"Introduces CONCH, the vision-language foundation model that achieves the highest accuracy in the study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Quilt-LLAVA, the 7B-parameter instruction-tuned model whose performance is compared against CONCH."},{"cited_title":"In: International conference on machine learning","cited_arxiv_id":null,"evidence_quote":"Defines the CLIP contrastive dual-encoder paradigm that Quilt-Net's architecture adapts for histopathology."},{"cited_title":"In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the hierarchical self-supervised pretraining approach that Quilt-Net builds on for gigapixel slides."},{"cited_title":"Nature Communications16(1), 1239 (2025)","cited_arxiv_id":null,"evidence_quote":"Documents prompt-sensitivity of VLMs in oncology, motivating the paper's systematic prompt ablations."}],"review_version":1}