{"id":"740f3f8e-0163-435e-a4f1-466906f44deb","arxiv_id":"2501.05409","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Atlas, a 632M-parameter ViT pathology model trained on 1.2M multi-stain slides, achieves a 61.9 percent average on 21 public benchmarks, the best among seven leading foundation models.","lead":"Atlas, a new pathology foundation model from Mayo Clinic, Charité, and Aignostics, was trained on 1.2 million hospital slides and achieves the highest average score across 21 public benchmarks (61.9 percent). The result is a practical advance in computational pathology, offering a strong tile encoder for tasks like cancer subtyping and biomarker prediction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA margin is not shown to be significant: per-task SDs exceed the 1.1 p.p. lead and the CLS-only average cuts it to 0.4 p.p.; need paired CI.","rationale":"The paper is transparent: it reports per-task standard deviations, gives CLS-only and CLS+Mean tables, and describes the evaluation protocol in detail. Those strengths make the central claim testable rather than hiding the issue. However, the headline result is a single unweighted average over heterogeneous benchmarks, and the reported uncertainty at the task level is large relative to the 1.1 p.p. margin. The better-of-two-readout rule is applied uniformly, so it is not unfair, but it makes the result protocol-dependent: switching to CLS-only reduces Atlas's lead over Virchow2 from 1.1 p.p. to 0.4 p.p. (60.6 vs 60.2). The absence of any aggregate confidence interval or paired significance test leaves the SOTA claim unsupported in the precise sense that a reader cannot tell whether the margin is signal or noise. The reader's weakest assumption identifies exactly this issue, and my proposed check would settle it directly. If the paired test survives, the claim stands; if not, the verdict should remain conditional and the claim softened. No other concern is as load-bearing: model release and training-data overlap are secondary because the benchmarks are public and the comparisons are internally consistent.","tokens_in":16238,"tokens_out":5134,"duration_ms":49120,"concrete_test":"From Table 3, compute paired per-benchmark differences (Atlas minus Virchow2 and Atlas minus H-Optimus-0) and run a paired bootstrap or Wilcoxon signed-rank test over the 21 tasks, optionally weighting by inverse squared standard error from the 5 seeds. Repeat with CLS-only results from Table 4. If the 95% CI for the mean difference includes 0 or the signed-rank p > 0.05, the SOTA claim should be downgraded to 'competitive within noise'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 4, Table 1) is that Atlas's 61.9% average beats Virchow2 and H-Optimus-0 (60.8%) by 1.1 p.p. The load-bearing assumption is that this aggregate margin is a real signal, not selection noise. Table 3 reports per-task standard deviations over 5 seeds that are often 2-8 p.p. for HEST tasks, several times the headline margin, and no confidence interval is given for the aggregate. The protocol reports max(CLS, CLS+Mean) per model (Section 3.3); Table 4 shows that with CLS-only Atlas averages 60.6, only 0.4 p.p. above Virchow2's 60.2 and below Virchow2's CLS+Mean 60.8. Thus the exact magnitude and even the identity of the best model depend on a readout-choice rule. The simple unweighted average also mixes Pearson correlations and balanced accuracies. Without a paired significance test across the 21 benchmarks, the 1.1 p.p. SOTA claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This report introduces Atlas, a 632M-parameter ViT-H/14 tile encoder for histopathology, trained on 1.2 million whole-slide images from Mayo Clinic and Charité using the RudolfV/DINOv2 paradigm. The authors evaluate Atlas against six public pathology foundation models on 21 public benchmarks spanning gene-expression prediction, MSI classification, TIL detection, cancer subtyping, tissue classification, and slide-level tasks, using the eva and HEST evaluation frameworks plus an internal linear-probing framework. The paper's central claim is that Atlas achieves a 61.9% average score, a 1.1 percentage-point improvement over Virchow2 and H-Optimus-0, despite being neither the largest model by parameter count nor the model trained on the most data.","tokens_in":16489,"tokens_out":6823,"duration_ms":65777,"significance":"If established, the result would be a practically useful reference point for pathology foundation models: it suggests that broad real-world data diversity (multiple stains, magnifications, scanners) can yield a strong tile encoder without maximal model or data scale. The evaluation has real strengths: twenty-one public benchmarks, external evaluation frameworks for most tasks, five-seed or five-fold repetitions with reported variation, direct comparison with six leading models, and a token-readout ablation in the appendix. The circularity risk is low because the headline numbers are measured on benchmarks not constructed by the authors. However, the headline margin is fragile and currently not statistically substantiated, so the main contribution is conditional on additional analysis.","major_comments":[{"comment":"The central claim that Atlas beats Virchow2 and H-Optimus-0 by 1.1 p.p. on average is not backed by a significance test. The per-task uncertainties in Table 3 are much larger than the aggregate margin: for example, HEST-IDC is 60.4±8.3 for Atlas versus 61.0±8.1 for H-Optimus-0, HEST-PAAD is 51.8±7.4 versus 50.9±4.3, and HEST-SKCM is 62.5±2.4 versus 66.1±5.8, the latter favoring H-Optimus-0 beyond the headline gap. Section 3.3 calls the reported values \"standard errors\" while Table 3's caption calls them \"standard deviation\"; either way, no confidence interval for the 21-task average difference is given. A paired bootstrap or Wilcoxon signed-rank test across the 21 tasks is needed before \"state-of-the-art\" is claimed.","section":"§4 and Table 1; Table 3"},{"comment":"The evaluation rule \"report the better (maximum) performance of the two\" for CLS and CLS+Mean tokens is load-bearing for the ranking. With CLS-only embeddings, Atlas averages 60.6% over all 21 tasks, only 0.4 p.p. above Virchow2 (60.2%), whereas with the max rule the lead is 1.1 p.p. and under CLS+Mean Virchow2 averages 60.8%, close to Atlas's 61.9%. Thus the magnitude of the advantage is sensitive to the readout-selection rule, and applying the max per task can inflate each model's score differently. The authors should either prespecify a single readout before seeing results or provide a paired comparison that treats the readout rule as part of the protocol and reports both the per-task selection and a fixed-readout sensitivity analysis.","section":"§3.3, Table 4"},{"comment":"The overall \"average performance score\" mixes Pearson correlations from the ten HEST regression tasks with balanced accuracies from the eleven classification tasks. These metrics are on different scales, and the unweighted average gives equal weight to high-variance tasks such as HEST-IDC and HEST-PAAD alongside near-deterministic tasks such as TCGA Uniform and CRC-100k. As a result, \"61.9%\" is not a directly interpretable performance measure, and the 1.1 p.p. difference could be driven by the choice of task weighting rather than by a consistent model advantage. The authors should report a paired difference analysis on a common scale (for example, rank-normalizing each task) or at least show the per-group averages with confidence intervals.","section":"§3.3, §4, Tables 1–3"},{"comment":"Several entries in Table 3 report zero standard deviation for tasks where fold-to-fold variation is expected, including HEST-COAD for Gigapath (30.7±0.0), HEST-PRAD for H-Optimus (38.5±0.0), and all TCGA Uniform rows, whereas other HEST rows exceed ±7 p.p. The caption says these are standard deviations over five data splits, but the zeros suggest either deterministic splits, a different number of runs, or a reporting artifact. The paper should clarify this so that the uncertainty information can actually be used to assess whether Atlas's edge is real.","section":"Appendix A.2, Table 3"}],"minor_comments":[{"comment":"The text says \"We report the mean performance and standard errors over the seeds\" while Table 3's caption says \"standard deviation over the 5 data splits\"; please align the terminology and state explicitly whether the reported ± values are standard errors, standard deviations, or something else.","section":"§3.3 and Table 3 caption"},{"comment":"The phrase \"state-of-the-art\" in the abstract and in Section 4 should be qualified until the significance analysis requested in my major comments is provided, since the point estimate alone does not establish superiority over the closest contenders.","section":"Abstract and Section 4"},{"comment":"The paper does not include a model or code availability statement; for a foundation model report, an explicit statement on whether the Atlas weights and evaluation code will be released would greatly aid reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the submission is a technical report with a commercially relevant model developed by Aignostics, Mayo Clinic, and Charité. The evaluation is broad and externally anchored, which is commendable, but the headline SOTA claim is currently a point-estimate artifact that depends on the token-readout rule and on an unweighted average of incommensurable metrics. I would ask for a paired statistical analysis and a prespecified readout rule before considering this for publication. The self-citation to RudolfV is understandable given that Atlas is built on that method, but the report would benefit from an explicit statement of which training choices are new beyond data scale and data diversity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a competent, transparent model report, and the central claim — Atlas is the best publicly benchmarked tile encoder on this 21-task suite — is probably true, but the headline 1.1 pp margin is fragile. The stress-test note is right that the margin is not shown to be significant: with CLS-only readout the margin shrinks to 0.4 pp, and the per-task standard deviations on HEST tasks are several times larger than the aggregate lead. Still, the paper earns credit for reporting all the numbers that make the fragility visible.\n\nWhat's new: a ViT-H/14 trained on 1.2M slides from two medical centers, with multi-stain and multi-magnification data. The architecture and training procedure come from RudolfV, so the novelty is in the scale and data diversity, not the method. The evaluation is broad: 21 public benchmarks spanning morphology and molecular tasks, using the eva and HEST frameworks, with 5 seeds and per-task SDs. They also report the one benchmark where Atlas is below average (TCGA Uniform 20x) rather than hiding it. That is the right kind of behavior for a model report.\n\nSoft spots: (1) no aggregate confidence interval for the 61.9% average, so we cannot say whether the 1.1 pp lead over Virchow2 is real; (2) the better-of-CLS/CLS+Mean selection inflates scores and Table 4 shows the CLS-only margin is 0.4 pp, which is negligible; the primary should have been a fixed readout, with max as sensitivity; (3) mixing Pearson correlations and balanced accuracies in an unweighted average is a questionable summary; (4) no weights released, so others cannot verify the embeddings on new tasks. These are all addressable and none of them makes me think the model is bad.\n\nWho is this for: the computational pathology community, anyone building on tile encoders. It deserves a serious referee, though a reviewer should ask for a paired test or at least a bootstrap CI on the delta, and for weights or an API. I'd send it to review rather than desk reject.\n\nRegards","headline":"Solid, transparent model report whose SOTA claim is plausible but not statistically pinned down — worth reviewing, but the 1.1 pp headline margin shrinks to 0.4 pp under CLS-only readout.","tokens_in":17111,"tokens_out":1939,"would_cite":true,"duration_ms":18085,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Atlas, a 632-million-parameter pathology model, tops 21 public benchmarks with a 61.9% average.","keywords":["pathology foundation model","self-supervised learning","DINOv2","ViT-H/14","whole slide images","linear probing","gene expression prediction","digital pathology"],"falsifier":"Re-run the full 21-benchmark evaluation under a single fixed token rule (CLS only, or CLS+Mean only) and compute a paired, seed-level significance test across tasks; if the Atlas advantage over Virchow2 and H-Optimus-0 shrinks to within one standard error or flips sign on several tasks, the state-of-the-art claim would fail. A simpler check: count how often Atlas wins when differences are required to exceed the reported per-task standard deviation.","tokens_in":37,"feed_emoji":"🔬","tokens_out":6311,"duration_ms":118660,"temperature":0.7,"pith_summary":"The paper introduces Atlas, a ViT-H/14 pathology tile encoder with 632 million parameters, trained on 1.2 million whole-slide images (about 3.4 billion tiles) from two hospital systems using the RudolfV self-supervised recipe built on DINOv2. The authors' central claim is that Atlas sets a new state of the art on a 21-dataset public benchmark suite, averaging 61.9% across molecular (gene-expression prediction) and morphology (tissue and cell classification) tasks, which is 1.1 percentage points above the two closest contenders, Virchow2 and H-Optimus-0. The result matters because Atlas is neither the largest model nor the one trained on the most slides, suggesting that training-data diversity (multiple stains, magnifications, and institutions) can compensate for raw scale. If the claim holds, it shifts attention from simply scaling parameters and data volume toward curation and sampling strategy for pathology foundation models.","feed_headline":"Pathology model beats larger rivals on 21 benchmarks","feed_subtitle":"A 632M-parameter encoder trained on 1.2M slides averages 61.9%, edging Virchow2 and H-Optimus-0 by 1.1 points.","key_machinery":"The engine of the result is Atlas itself: a ViT-H/14 vision transformer with 632 million parameters, trained with the RudolfV paradigm, a DINOv2-based self-supervised approach, on roughly 520 million sampled tiles drawn from 1.2 million whole-slide images covering more than 70 tissue types, over 100 staining types, and seven scanners across two institutions. The training tiles are extracted at four magnifications (0.25, 0.5, 1.0, and 2.0 microns per pixel), and the corpus includes H&E, immunohistochemistry, and special stains. The evaluation machinery is the linear-probing protocol: for each frozen backbone, both the CLS token and the CLS+Mean token (CLS concatenated with the mean of patch tokens) are probed and the better score is kept, with tasks run under the eva and HEST frameworks plus an internal logistic-regression pipeline.","core_discovery":"On its own terms, the paper claims that Atlas achieves state-of-the-art performance across twenty-one public benchmark datasets, with an average score of 61.9%, a 1.1 percentage point improvement over the two closest contenders, Virchow2 and H-Optimus-0, both at 60.8%. Atlas leads on 11 of the 21 tasks, is second-best on 7 of the remaining 10, and is below average on a single benchmark, TCGA Uniform (20×). The margin is concentrated in morphology tasks, where Atlas averages 84.6% versus Virchow2's 84.0%, while on molecular tasks Atlas (44.9%) and H-Optimus-0 (44.8%) are effectively tied. The evaluation uses frozen-backbone linear probing, reporting the better of CLS-token and CLS+Mean-token embeddings for every model, with five seeds per task.","pith_inferences":["Because the reported margin is smaller than per-task standard deviations, a seed-paired significance test could show that Atlas and its closest rivals are statistically indistinguishable; that test is not in the paper.","Atlas's one clear failure, TCGA Uniform (20×), hints at magnification-specific sensitivity; a testable extension is training or fine-tuning with magnification-conditioned tokens.","The two-institution corpus raises an open generalization question: whether the advantage persists on slides from scanners and staining protocols outside Mayo Clinic and Charité.","The better-of-CLS/CLS+Mean rule inflates every model's score, but it may not inflate them equally; reporting the chosen token per task would make leaderboards more reproducible."],"forward_implications":["If the claim holds, pathology foundation models do not need to maximize parameters or slide count; a mid-sized model trained on a diverse mid-sized corpus can outscore both a larger-parameter model and a larger-data model on average.","Multi-stain, multi-magnification training transfers to both morphology and molecular tasks, making Atlas a strong single encoder for TIL detection, cancer typing, and gene-expression prediction.","The 21-task average of 61.9% becomes a reference number that future pathology foundation-model papers will need to beat, and the per-task table provides a granular checklist for where gains come from.","Clinically oriented downstream applications, such as MSI detection and Gleason grading, inherit whatever gains the encoder provides, since those tasks are among the 21 benchmarks."],"supporting_citations":[{"why":"Supplies the RudolfV training paradigm on which Atlas is based.","marker":"[10]"},{"why":"Provides the DINOv2 self-supervised framework underlying the training approach.","marker":"[30]"},{"why":"Supplies the eva evaluation framework used for linear probing and slide-level ABMIL tasks.","marker":"[14]"},{"why":"Defines the HEST-1k benchmark and its gene-expression regression protocol used for molecular tasks.","marker":"[16]"},{"why":"Virchow2 is the closest contender trained on more slides and a key baseline for the SOTA claim.","marker":"[42]"},{"why":"H-Optimus-0 is the closest contender with the largest parameter count and a key baseline.","marker":"[34]"},{"why":"Prov-GigaPath provides the slide-based model whose tile encoder is compared as a top contender.","marker":"[41]"},{"why":"UNI is a widely used tile-based pathology foundation model included as a baseline.","marker":"[7]"},{"why":"Phikon-v2 is a public feature extractor baseline in the comparison.","marker":"[13]"}],"fun_headline_variants":["Atlas pathology model tops 21 benchmarks with 61.9%","Smaller model Atlas edges Virchow2, H-Optimus-0 by 1.1 points","Morphology strength lifts Atlas to state-of-the-art on 11 tasks","Atlas: 632M params, 1.2M slides, best average on 21 datasets","Atlas leads 11 of 21 tasks, ties molecular tasks with rivals"],"cache_read_input_tokens":19200,"weakest_assumption_plain":"The headline claim depends on the 1.1-percentage-point average edge over Virchow2 and H-Optimus-0 being a real effect rather than evaluation noise, since per-task standard deviations across the five seeds are often several times larger than that margin.","fun_headline_variants_meta":{"raw":{"variants":["Atlas pathology model tops 21 benchmarks with 61.9%","Smaller model Atlas edges Virchow2, H-Optimus-0 by 1.1 points","Morphology strength lifts Atlas to state-of-the-art on 11 tasks","Atlas: 632M params, 1.2M slides, best average on 21 datasets","Atlas leads 11 of 21 tasks, ties molecular tasks with rivals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000148,"raw_usage":{"total_tokens":1139,"prompt_tokens":842,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":184}},"tokens_in":458,"tokens_out":297,"duration_ms":3079,"temperature":1.0,"reasoning_tokens":184,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:12:45.033348+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full 21-benchmark evaluation under a single fixed token rule (CLS only, or CLS+Mean only) and compute a paired, seed-level significance test across tasks; if the Atlas advantage over Virchow2 and H-Optimus-0 shrinks to within one standard error or flips sign on several tasks, the state-of-the-art claim would fail. A simpler check: count how often Atlas wins when differences are required to exceed the reported per-task standard deviation.","supporting_citations":[{"cited_title":"eva: Evaluation framework for pathology foundation models","cited_arxiv_id":null,"evidence_quote":"Supplies the eva evaluation framework used for linear probing and slide-level ABMIL tasks."},{"cited_title":"Song, Ming Y","cited_arxiv_id":null,"evidence_quote":"Defines the HEST-1k benchmark and its gene-expression regression protocol used for molecular tasks."},{"cited_title":"H-optimus-0","cited_arxiv_id":null,"evidence_quote":"H-Optimus-0 is the closest contender with the largest parameter count and a key baseline."},{"cited_title":"Wright, Ari Robicsek, Brian Piening, Carlo Bifulco, Sheng Wang, and Hoifung Poon","cited_arxiv_id":null,"evidence_quote":"Prov-GigaPath provides the slide-based model whose tile encoder is compared as a top contender."},{"cited_title":"Chen, Tong Ding, Ming Y","cited_arxiv_id":null,"evidence_quote":"UNI is a widely used tile-based pathology foundation model included as a baseline."}],"review_version":1}