{"id":"ee65bf10-f613-409d-88c5-9e828a50a093","arxiv_id":"2607.19261","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PathAgentBench shows current pathology vision-language models can integrate supplied multi-scale findings but largely fail at acquiring diagnostic evidence directly from whole-slide images.","lead":"PathAgentBench is a new benchmark that tests whether AI pathology models can find diagnostically important regions in whole-slide images, not just reason about regions they are given. Across 20 vision-language models, reasoning about supplied evidence was strong (over 93% on one task), while actively locating evidence was very weak.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mode B's fixed hierarchical pipeline and 5% overlap rule may conflate a model's evidence-acquisition failure with the benchmark's own pruning structure; a re-scoring-only test would settle whether the 2.02% high-mag hit rate reflects model capability or protocol.","rationale":"The reader's weakest_assumption focuses on the single pathologist ground truth (diagnostic trees and tumor polygons) and inter-annotator reliability, which is a valid concern but not the most load-bearing one: if a second pathologist would annotate a slightly different tree/polygon, the absolute numbers shift but the qualitative asymmetry (high T1/T2/T4, low T3) would likely persist because the same annotations are used throughout. My concern is more direct about the interpretation of the Mode B numbers: the benchmark's central claim, stated in the abstract and Finding 3, is that current VLMs fail at evidence acquisition from gigapixel WSIs. But Mode B's success metric is completely entangled with the fixed top-K hierarchical protocol. The oracle sweep in Table IV is decisive evidence of this: even with ground-truth knowledge of which tiles are tumor (the oracle presumably ranks positive tiles perfectly), unconditional hit rate collapses from 67.11% at 2.5x to 3.01% at 40x under the default schedule. The protocol's irrevocable pruning is so aggressive that almost no positive tile remains reachable, regardless of the scorer. Therefore the 2.02% figure for Gemini is not a clean measurement of 'evidence acquisition by a VLM'; it is the output of a specific search algorithm with fixed branching factors. This does not destroy the paper's broader point—Mode A also shows poor localization, and the Mode B conditional hit rates do decline with magnification (Gemini 46.81→22.50→7.94), which is model-dependent and not an artifact—but it does mean the headline narrative 'evidence acquisition is the primary bottleneck' is partly pre-determined by the protocol. My proposed test would isolate model scoring capability from search-structure attrition. I therefore keep the reader's CONDITIONAL verdict but frame the condition differently: rather than requiring only annotation reliability, I require either a no-pruning control or a re-reporting of average precision per level. The reader identified a real concern (inter-annotator reliability), and I partially agree, but the strongest threat is the protocol-metric confound. I do not see internal inconsistency, cheating, or dishonest reporting; the paper's own oracle sweep actually makes the confound visible. The test I propose is feasible within the released benchmark (though the private cohort is a limitation) and would definitively settle whether the headline numbers are a model-capability statement or a search-protocol statement.","tokens_in":18121,"tokens_out":2542,"duration_ms":24226,"concrete_test":"Re-run Mode B on the same 190-slide breast cohort with the same per-tile scoring prompts, but remove the hierarchical top-K retention: score all tiles independently at each of 2.5x/10x/40x and compute micro-averaged unconditional hit rates per magnification for Gemini-3-Flash and Qwen-3.5-Flash. Compare against the reported 52.2/18.5/2.02 and 51.75/18.46/2.06. Also run the oracle scorer (ground-truth tumor polygon as the 'score') under the same no-pruning protocol. If the no-pruning 40x hit rate is substantially higher than 2%, the original protocol's pruning—not the model's acquisition capability—is the dominant cause of the headline decay.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central finding — a pronounced gap between evidence reasoning and evidence acquisition — rests heavily on Task 3 Mode B, because Mode A is evaluated on only 4 models and the private cohort lacks shared-path annotations (Section III-B). In Mode B, the VLM never plans: it only scores tiles inside a fixed top-K pipeline (Section III-D3). The headline numbers 52.2% → 18.5% → 2.02% are unconditional hit rates micro-averaged over the cohort, so they are not pure measures of the VLM's ability to recognize tumor at high magnification; they compound three protocol decisions: the retention schedule (6,3,2), the irrevocable pruning, and the scoring prompt. The cited Table IV oracle (67.11/24.68/3.01 at default) demonstrates that even a ground-truth scorer suffers the same collapse, dropping from ~67% to ~3%. Thus the steep decrease is partly an artifact of the hierarchical search structure itself, not solely a property of VLM acquisition skill. Moreover, the 5% tumor-overlap rule (Section III-D3) can mislabel tiles containing small but diagnostically relevant tumor fractions as negative, and no inter-annotator agreement is reported for the tumor polygons. If the protocol, rather than model capability, drives the decay, then 'evidence acquisition is the primary bottleneck' is overstated as a model deficiency: the benchmark would instead show that any scorer, including an oracle, loses recall under aggressive top-K pruning. A concrete check: run Mode B with the VLM as a per-tile scorer on all tiles at each magnification (no top-K pruning); compute unconditional hit rates at each level directly. If the unpruned hit rate at 40x is high (e.g., >50%), the original protocol's irrevocable pruning is the dominant cause of the 2.02%, and the bottleneck claim needs rephrasing. Conversely, if unpruned hit rates still collapse, the finding is robust. This is the single most load-bearing concern because it determines whether the benchmark measures evidence acquisition or merely","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"PathAgentBench is a new benchmark for whole-slide pathology image analysis that decomposes 'evidence seeking' into four tasks: T1 image-to-text matching, T2 text-to-image retrieval, T3 diagnostic-region localization (Mode A: text-guided; Mode B: autonomous whole-slide exploration), and T4 multi-scale diagnostic reasoning. The data are 1,822 TCGA WSIs with 17,135 pathologist-authored diagnostic paths and a private 190-slide breast cohort with annotated tumor polygons. The authors evaluate 20 VLMs (general-purpose, medical, pathology-specialized) plus non-VLM baselines. Their main empirical claim is that current VLMs are strong at interpreting and integrating supplied evidence (T4 > 93% for leading models, T1/T2 around 50–68%) but fail at acquiring evidence directly from WSIs (Mode A mIoU < 0.09; Mode B unconditional hit rate falls from 52.2% to 2.02% across 2.5×/10×/40×). They conclude that evidence acquisition, not reasoning, is the primary bottleneck for pathology agents.","tokens_in":18495,"tokens_out":7497,"duration_ms":72413,"significance":"The resource contribution is substantial: 17,135 annotated diagnostic paths, 51,363 node findings, four complementary tasks, an expert human reference, and evaluations across 20 model configurations with several ablations (distractor encoders, budget sweeps, oracle baseline, per-organ breakdown). The task decomposition is a genuine advance over patch-only or slide-level benchmarks because it makes the acquisition-versus-reasoning distinction measurable. The central finding, if supported, would have clear impact on how pathology-VLM progress is reported. However, as detailed below, the protocol-level controls—especially the oracle row in Table IV and the absence of annotation-uncertainty analysis—currently leave the size and even the direction of the model-specific acquisition failure less certain than the paper claims. The benchmark framework itself remains useful and likely citable.","major_comments":[{"comment":"The oracle row in Table IV is the key control, and it undercuts the headline interpretation. Under the default (6,3,2) schedule, a ground-truth oracle gets 67.11/24.68/3.01 unconditional hit rate at 2.5/10/40×, whereas Gemini gets 54.82/19.89/2.24 and Qwen 51.75/18.46/2.06. Thus even perfect per-tile tumor classification cannot recover from the fixed hierarchical pruning: high-magnification recall is capped near 3%. The reported 52.2%→2.02% decay therefore reflects the search protocol as much as VLM capability, and Finding 3's 'evidence acquisition remains the primary bottleneck' conflates protocol with model. Please report a normalized metric (VLM unconditional hit rate divided by oracle hit rate under the same schedule, or conditional recall among tiles actually scored) and discuss where the model-specific loss is concentrated. Without this, the paper overstates the model deficiency.","section":"§IV-G, Table IV"},{"comment":"Mode B's tile-level ground truth is defined by more than 5% area overlap with a single clinically annotated tumor polygon. No inter-annotator agreement is reported for these polygons. If the polygon under-segments or over-segments the tumor, or if diagnostically relevant tumor foci occupy less than 5% of a tile, the VLM's valid acquisitions are scored as misses. Since Mode B is the main evidence for the acquisition gap (Mode A is run on only four models), this labeling uncertainty is load-bearing. Report inter-annotator agreement on a subset of the private cohort and a sensitivity analysis of the 5% threshold (e.g., 0, 1, 5, and 10%).","section":"§III-D3"},{"comment":"Mode A is evaluated on only three VLMs (GPT-5.2, Gemini-3-Flash, Qwen-3.5-Flash) plus Patho-R1-7B; other open-weight models were excluded for failing to emit valid bounding-box tool calls. With 150 queries per model on a 50-slide subset, the claim that current VLMs are poor at text-guided localization is based on a self-selected, tool-capable subset. The comparison to non-VLM heuristics is informative, but the paper should explicitly restrict its conclusion to 'tool-capable models evaluated' or add more tool-capable checkpoints; otherwise the breadth of the 'all models' claim (Abstract, Finding 3) exceeds the evidence.","section":"§III-D3, Table III"},{"comment":"The TCGA diagnostic trees are single-path annotations: one annotator selects root-to-leaf paths, and a supervisor reviews. No quantitative inter-annotator agreement is reported. In Tasks 1–4 and Mode A, any model-selected region or finding that does not match the annotated path is counted as a miss. If pathologists differ in their chosen diagnostic routes, the benchmark may undercount valid evidence acquisition and overstate the acquisition gap. At minimum, report path-level agreement among annotators on a subset (e.g., kappa on selected regions, or agreement on findings) and discuss whether the single-path supervision affects the headline numbers.","section":"§III-C"}],"minor_comments":[{"comment":"The author block and abstract contain spacing artifacts ('Y ufeng Wu', 'Y ueming Jin'); please fix.","section":"Abstract / Author block"},{"comment":"Fig. 5 legend ('Qwen3.5-Flash35B/3B activate') and Fig. 6 caption ('Scale EncodingSelected at 2.5x') appear garbled in the compiled PDF; check typography.","section":"Fig. 5 / Fig. 6"},{"comment":"The T4 claim that open-weight ≤26B models beat closed-source VLMs should be scoped as 'in the text-only, supplied-evidence MCQ setting'; otherwise it could be misread as end-to-end superiority.","section":"§IV-D"},{"comment":"Ensure the '—' entries are consistent with the stated 20-model evaluation. For example, Mode B is described as '15 successfully evaluated models,' but the table lists more model rows; clarify the inclusion criterion.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is likely suitable for the journal if the authors tighten the interpretation. The most important fix is to use the oracle control to separate protocol-induced attrition from model capability. I would also encourage the authors to report inter-annotator agreement and threshold sensitivity, since the private cohort's annotations are not public. No concerns about originality or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nPathAgentBench is worth your time. The four-task decomposition — interpretation, verification, acquisition, integration — cleanly separates reasoning over supplied evidence from acquiring it, and the authors did the hard work of building a 1,822-WSI, 17,135-path dataset with pathologist-authored diagnostic trees plus a private 190-slide breast cohort for navigation. The distractor-encoder ablation (MiniLM vs BiomedCLIP vs CONCH) that preserves model ranking is exactly the kind of sanity check benchmarks should ship with.\n\nThe core finding, that current VLMs integrate supplied evidence much better than they acquire it, is directionally robust. Mode A closes at mIoU <0.09 while a parent-box-center heuristic gets 0.25–0.28, and that gap holds across the few tool-capable models. The paper earns its conclusion that evidence acquisition is the bottleneck.\n\nThe soft spot is Mode B. The paper's headline number — unconditional hit rate falling 52.2% → 18.5% → 2.02% — is presented as a model deficiency, but the protocol does the pruning: a VLM scores tiles, top-K retention is irrevocable, and even the ground-truth oracle collapses from 67.11% to 24.68% to 3.01% under the same schedule. So the absolute numbers are as much about the search structure as about VLM capability. The authors do acknowledge this coupling in the ablations, but the discussion still frames acquisition as the 'primary bottleneck' without cleanly separating the two. A re-scoring-only test — scoring all tiles at each magnification, no pruning — would settle whether the 2.02% reflects model acquisition skill or protocol. As is, the qualitative claim is plausible, the exact figures are fragile.\n\nOther soft spots are known: Mode A runs only on four tool-capable models; annotation reliability isn't reported (one annotator per slide plus supervisor review, but no inter-annotator agreement); the 5% tumor-overlap rule could mislabel small but relevant tumor fractions; the private cohort and benchmark artifacts aren't available with a verifiable identifier. None of these overturn the central asymmetry, but they make the numeric values provisional.\n\nWho is it for: anyone building pathology agents or gigapixel-scale evaluation. The framework transfers to other imaging domains with hierarchical navigation. Accept for peer review; require artifact release with commit hash, inter-annotator stats, a Mode B no-pruning control, and sensitivity analysis on the overlap threshold before treating the numbers as community standard.\n\nBest.","headline":"A genuinely useful benchmark with a real finding, but the Mode B headline hit-rate numbers are half protocol artifact; still worth engaging seriously.","tokens_in":19095,"tokens_out":1913,"would_cite":true,"duration_ms":19427,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pathology vision-language models can interpret supplied evidence but fail to find it in whole-slide images.","keywords":["whole-slide images","pathology vision-language models","evidence seeking","diagnostic tree","benchmark","region localization","multi-scale reasoning","agentic exploration"],"falsifier":"Re-run the autonomous exploration with an oracle-guided backtracking policy that re-opens pruned branches at high magnification; if unconditional 40× hit rate remains below roughly ten percent, the bottleneck is perceptual or score quality, whereas a jump above thirty percent would show that the reported gap is recoverable search rather than intrinsic acquisition failure.","tokens_in":17988,"feed_emoji":"🔬","tokens_out":5215,"duration_ms":53396,"temperature":0.7,"pith_summary":"The paper argues that whole-slide pathology diagnosis is an evidence-seeking process—deciding where to look, zooming in, and integrating findings—and that current vision-language models are strong at the final reasoning step but weak at the earlier searching steps. To make this measurable, it builds a diagnostic-tree benchmark with 1,822 slides and 17,135 pathologist-authored diagnostic paths, scored across four tasks: image-to-text matching, text-to-image retrieval, region localization, and multi-scale reasoning. Results show leading open-weight models exceed 93% accuracy on multi-scale reasoning and over 50% on matching tasks, but the best text-guided localization reaches below 0.09 mean IoU—worse than a simple center-based heuristic—and autonomous exploration loses coverage from 52% at low magnification to 2% at high magnification. The paper concludes that evidence acquisition, not evidence interpretation or integration, is the primary bottleneck holding pathology agents back from end-to-end use. This matters because current patch- and slide-level benchmarks, which supply evidence to the model, overstate how ready these systems are for real slides.","feed_headline":"Best pathology AI scores below 0.09 on locating diagnostic regions","feed_subtitle":"Models excel when evidence is supplied, then collapse when asked to navigate gigapixel slides for it.","key_machinery":"The diagnostic tree: each whole-slide image is a tree whose root is the slide thumbnail and whose nodes are regions at 2.5×, 10×, and 40× magnifications linked by pathologist-annotated parent-child relations; a root-to-leaf path is one diagnostic route. The benchmark formalizes diagnosis as marginalizing over latent search paths, separating the evidence-reasoning term from the evidence-acquisition term. Four tasks operationalize this decomposition: image-to-text matching, text-to-image retrieval, diagnostic-region localization (text-guided and autonomous-exploration modes), and multi-scale diagnostic reasoning. The two localization modes isolate complementary failure modes—planning and tool","core_discovery":"On the paper's own terms, the central discovery is a capability asymmetry: once multi-scale findings are supplied, models integrate them into a diagnosis almost as well as pathologists, but when asked to localize or autonomously acquire the same evidence from a gigapixel slide, every evaluated model collapses. In text-guided localization, the best mean intersection-over-union is below 0.09, underperforming a parameter-free parent-box center baseline; in autonomous exploration, unconditional tumor hit rate falls from 0.522 at 2.5× to 0.185 at 10× to 0.020 at 40× under a fixed retention schedule. The paper therefore claims that current pathology VLMs are useful as scorers of given tiles, not a","pith_inferences":["The text-to-image retrieval advantage over image-to-text matching points to hypothesis-driven search as a natural agent loop: a model that verifies better than it describes should generate differential hypotheses and retrieve evidence to test them, rather than scan without a hypothesis.","Because the diagnostic-tree ground truth comes from a single annotator per slide with review, the measured acquisition gap may be partly annotation-dependent; collecting inter-annotator agreement on diagnostic paths would separate model failure from legitimate variation in pathologist search behavior.","The paper's own budget sweep shows that adding branching recovers recall without yielding precise localization; a natural extension is an oracle-bounded study that measures how much of the high-magnification coverage loss is reachable even with perfect tile scoring.","The four-task decomposition could transfer to other gigapixel-imaging domains—such as radiology or satellite and document imaging—where the bottleneck is also deciding where to look across scales, not just interpreting what is shown."],"forward_implications":["Slide-level leaderboards that supply evidence to the model overstate end-to-end readiness; reasoning accuracy alone does not imply the ability to find the regions.","Agent design should prioritize backtracking and error recovery, since greedy hierarchical pruning is irrevocable and early mistakes foreclose whole subtrees.","Reliable confidence ranking is a structural prerequisite for top-K tile retention; near-uniform scores cannot support pruning decisions.","Hybrid verification, pairing a VLM navigator with classical tissue classifiers or task-specific detectors, could veto wrong child selections without requiring the VLM to solve planning alone.","Efficient, privacy-preserving deployment must be part of the agenda, since iterative acquisition dominates latency and cost and repeatedly transmits slide tiles to external services."],"fun_headline_variants":["Pathology AI: 93% reasoning, <0.09 IoU when locating regions","Best pathology AI misses tumor regions: IoU under 0.09","AI pathologists collapse when asked to find evidence in slides","From 0.522 to 0.020: AI hit rate plummets at high zoom","Pathology VLMs reason well but can't locate evidence"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that pathologist-authored diagnostic trees and a single annotated tumor polygon are the unique ground truth for diagnostically relevant evidence; a tile counts as positive only when more than 5% of its area overlaps that one polygon, and the paper reports no inter-annotator agreement.","fun_headline_variants_meta":{"raw":{"variants":["Pathology AI: 93% reasoning, <0.09 IoU when locating regions","Best pathology AI misses tumor regions: IoU under 0.09","AI pathologists collapse when asked to find evidence in slides","From 0.522 to 0.020: AI hit rate plummets at high zoom","Pathology VLMs reason well but can't locate evidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000496,"raw_usage":{"total_tokens":2326,"prompt_tokens":861,"completion_tokens":1465,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":1375}},"tokens_in":605,"tokens_out":1465,"duration_ms":11783,"temperature":1.0,"reasoning_tokens":1375,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T12:54:22.569098+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the autonomous exploration with an oracle-guided backtracking policy that re-opens pruned branches at high magnification; if unconditional 40× hit rate remains below roughly ten percent, the bottleneck is perceptual or score quality, whereas a jump above thirty percent would show that the reported gap is recoverable search rather than intrinsic acquisition failure.","supporting_citations":[],"review_version":1}