Pith. sign in

REVIEW 2 major objections 5 minor 23 references

The paper establishes that raw Fréchet distances in pathology are encoder-dependent — a roughly thirty-fold spread for the same comparison — and shows that dividing each distance by the encoder's own within-cohort floor restores comparabili

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:50 UTC pith:OKCCUOBD

load-bearing objection Solid, reproducible benchmark of encoder dependence in FID-like pathology scores; the ratio-normalization fix is plausible but rests on an unproven floor-stability assumption. the 2 major comments →

arxiv 2607.20584 v1 pith:OKCCUOBD submitted 2026-07-22 eess.IV

HistoFID- Calibrating Frechet-distance evaluation across pathology foundation models

Swapnil Bhat , Devansh Lalwani , Maulik Shah , Sheena Alphones , Rimlee Dutta , Nikita Mulchandani , Roshani Gala , Jay Mehta This is my paper
classification eess.IV
keywords Fréchet distancecomputational pathologygenerative evaluationfeature extractionnormalizationhistology foundation modelsdistribution shiftslide-level aggregation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that a standard generative-evaluation score, the Fréchet distance, is not a stable quantity in computational pathology: for the same pair of tile sets, six feature encoders return values spanning roughly a factor of thirty, and the ordering does not follow embedding dimension. To fix this, the authors propose normalizing each measured distance by the encoder's own within-cohort floor — the distance between two disjoint halves of the same reference cohort. This ratio cancels per-encoder feature scale and cuts the across-encoder coefficient of variation by about 89% within a cohort and 58% across cohorts. After normalization, encoders separate into a sensitive group and an invariant group, and this split can change which generative model looks more realistic. A sympathetic reader would care because any pathology study using raw Fréchet distances with a histology foundation model is currently drawing conclusions that depend on the encoder choice.

Core claim

The central claim is that the raw Fréchet distance between two histology image sets is a property of the feature encoder as much as of the images, so it cannot be read or compared without naming the encoder. The paper demonstrates this by computing the distance for one fixed comparison with six encoders and observing a roughly thirty-fold spread, with no monotone relation to embedding dimension. It then shows that the ratio of each measured distance to the encoder's own floor distance — computed between two disjoint halves of the same cohort at matched sample size — makes the encoders largely agree: the across-encoder coefficient of variation drops from 1.19 to 0.13 within cohort and from 0.

What carries the argument

The load-bearing object is the ratio r_e = FD_test^e / FD_floor^e, where FD_floor^e is the Fréchet distance between two disjoint, sample-size-matched halves of the same reference cohort computed in encoder e's feature space. The floor captures the value that finite-sample estimation alone produces on that encoder, and since both numerator and denominator carry the same per-encoder feature scale, the scale cancels and a dimensionless, roughly encoder-invariant number remains. Sample sizes are matched at 25,000 per side to remove sample-size bias as a confound, and the coefficient of variation of the six per-encoder ratios quantifies agreement.

Load-bearing premise

The normalization assumes the within-cohort floor distance is a stable per-encoder baseline that reflects only finite-sample estimation noise and feature scale; if that floor shifts with tissue composition, staining, or cohort split, the ratio will inject a cohort-dependent denominator instead of canceling the encoder's scale.

What would settle it

Take one encoder and one cohort, split the reference into many disjoint halves, and recompute the floor distance on each split; if the floor varies by more than a small fraction of typical test distances, the ratio's cancelation claim fails. A second check: find one comparison where, even after normalization, two encoders rank two generative models in opposite order — the paper already shows encoder-dependent rankings on raw scores, so this would test whether normalization fully removes the dependence.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A raw Fréchet distance reported without naming its encoder and its floor is not interpretable; pathology studies using such a number are reporting an encoder-dependent quantity.
  • For detecting scanner drift, cohort differences, or stain shifts, a sensitive encoder is the right tool; for evaluating models where those nuisances should be ignored, an invariant, highly augmented self-supervised encoder is more appropriate.
  • Generative-model evaluation in pathology should report normalized distances under more than one encoder together with recall, because the encoder choice can change the ranking of which model is more realistic.
  • Slide-level distances from attention-pooling aggregators can detect per-slide composition changes that pooled patch-level distances are blind to, so tile-layout information is a separate axis of evaluation.
  • Under the same protocol, a proprietary pathology codec called TuroCompress attains the smallest file size and highest fidelity among the codecs tested.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The floor-ratio idea likely generalises beyond Fréchet distance: any feature-space distance whose magnitude scales with the encoder's feature scale — including sliced-Wasserstein distance or maximum mean discrepancy — should become more comparable under the same normalization, though the paper does not test this.
  • A natural next experiment is to bootstrap the reference split many times and check whether the floor distance is stable; if the floor shifts with tissue composition or cohort split, the denominator injects cohort-dependent signal rather than pure scale cancellation.
  • The consistent two-tier grouping suggests pathology evaluation papers should state which tier their encoder belongs to, or report results with one encoder from each tier, as a routine practice.
  • The slide-level result implies that generative whole-slide models need slide-level evaluation, since pooled patch distances ignore tile layout and can miss the very composition that a whole-slide generator is supposed to model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper quantifies how the Fréchet distance between fixed tile sets depends on the choice of feature encoder in digital pathology. Using six encoders (Inception-v3, Phikon-v2, CONCH, UNI2-h, Virchow2, Prov-GigaPath) and two cohorts (an in-house breast cohort and TCGA-BRCA), it reports a roughly thirty-fold spread in raw Fréchet distances for an identical within-cohort comparison. It then proposes a normalization: divide each distance by the encoder's own within-cohort floor, defined as the distance between two disjoint halves of the same cohort at matched sample size. The paper reports that this ratio reduces the cross-encoder coefficient of variation by about 89% within cohort and 58% across cohorts, and that encoders separate into two groups ('sensitive' and 'invariant') that are consistent across cross-cohort drift, stain shift, and generative evaluation. Additional results cover perturbation sensitivity, compression codecs including a proprietary codec TuroCompress, slide-level aggregation, and pathologist concordance.

Significance. If the normalization claim holds, the paper addresses a real and underappreciated problem: raw Fréchet distances computed with different pathology foundation models are not comparable, and encoder choice can change conclusions about generative models. The direct measurement of the raw spread across encoders is plausible and is a useful contribution, as is the release of the protocol and feature extracts. The paper also makes a good-faith effort to validate the normalization on independent comparisons (cross-cohort drift, perturbations, generative models). However, the central claim rests on the stability and interpretation of the within-cohort floor, and the manuscript currently does not provide the needed evidence. The main findings are therefore conditional on additional experiments and clarifications.

major comments (2)
  1. [Sec. 4.2, Fig. 2] The 'within-cohort' 89% CV reduction is not tied to a clearly defined test distance. The only within-cohort distance defined in Sec. 3.3 is the floor itself; if the distance being normalized in the 'within-cohort' claim is the floor, the ratio is exactly 1 for every encoder and the reduction is a tautology. The reported post-normalization CV of 0.13 is not 0, so the manuscript must be using a different within-cohort comparison, but that comparison is not specified anywhere. Please state exactly which distances enter the within-cohort CV calculation, provide the raw and normalized values, and add bootstrap or other confidence intervals for the 89% and 58% reductions. Without this, the headline quantitative claims cannot be evaluated.
  2. [Sec. 4.3, Table 3] The two-group separation and the subsequent conclusions in Sec. 4.7 (generative evaluation) and Sec. 5 depend on the ratio values in Table 3. Because the denominator in Eq. (4) is unvalidated as a stable per-encoder scale constant (see comment on Sec. 3.3), the grouping is not yet established. In particular, an unstable floor that varies with tissue composition or stain panel could make some encoders appear artificially 'sensitive' and others artificially 'invariant.' The manuscript should show that the grouping is robust when the floor is recomputed on different splits and when an alternative normalization, such as a common reference cohort floor, is used.
minor comments (5)
  1. [Sec. 3.3, Eq. (2)] The bias expansion E[FD_N] = FD_inf + c/N is stated without derivation or a precise statement of conditions. A brief justification or a citation beyond [4] would help.
  2. [Sec. 4.7, Table 6] The text says PixCell's recall is 'near zero,' but the Inception-v3 row reports 0.072. Please qualify the statement or explain why 0.072 is considered near zero in context.
  3. [Sec. 4.9] The pathologist concordance reports Spearman rho and p<0.05 but does not state the number of fields or readers per condition. Please add sample sizes and a description of the blinding/randomization protocol.
  4. [Tables 5 and 7] There are formatting errors: Table 5's TuroCompress row reads '6.555×>0.95' and Table 7's slide/patch ratio reads '3221.0'; these should be '6.5, 55×, >0.95' and '322 / 1.0' respectively.
  5. [Sec. 3.4, Eq. (12)] The min-max normalization in the perturbation panel makes each encoder's most disruptive perturbation equal to 1 and least equal to 0. This is clearly stated, but the caption of Table 4 could repeat that the absolute values are not comparable across encoders; currently it only says 'normalized within each encoder.'

Circularity Check

1 steps flagged

The ratio normalization is partly self-referential in its within-cohort demonstration, but the cross-cohort, perturbation, generative, and pathologist comparisons provide independent content, so the circularity is moderate.

specific steps
  1. self definitional [Sec. 3.3 Eqs. (3)-(4); Sec. 4.1-4.2, Table 2]
    "For the identical within-cohort comparison the raw distance ranged from 5.9 to 175.2 across the six encoders (Table 2) ... Dividing each distance by the encoder's own floor reduced the coefficient of variation across encoders from 1.19 to 0.13 within cohort (about 89%). Table 2: Intrinsic distance floor (reference half versus reference half)."

    The within-cohort comparison used for the 89% CV reduction is the floor itself: Eq. (3) defines the floor as FD_e(Phi_e(R_A), Phi_e(R_B)) and Sec. 4.1 identifies that value as the 'identical within-cohort comparison.' If the numerator in Eq. (4) is this floor, r_e = 1 for every encoder by construction, and the CV drops to (near) zero by dividing each value by itself. This makes the headline within-cohort normalization evidence arithmetic rather than empirical. The across-cohort 58% reduction, the perturbation panel, and generative/pathologist comparisons remain independent and carry the paper's empirical weight.

full rationale

The raw thirty-fold spread across encoders is a direct, credible measurement, and the paper's external checks (TCGA cross-cohort, controlled perturbations, generative-model ranking, blind pathologist concordance) are not derived from the normalization. The main circularity concern is that Eq. (4)'s ratio divides by a per-encoder floor defined in Eq. (3), so the within-cohort CV reduction reported in Sec. 4.2 is close to an identity if the test distance is the floor. Additionally, Eq. (3) asserts that the floor 'captures the value produced by finite-sample estimation alone' without testing floor stability across repeated splits or the mixed H&E/IHC composition of the in-house cohort; if the floor contains real cohort composition, the sensitive/invariant grouping becomes denominator-driven. That is a validity risk rather than an equation-level reduction, so it does not push the score higher. The paper is not fully circular: the ratio is an explicit normalization, and several headline conclusions are benchmarked against independent comparisons.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim depends on the assumption that a within-cohort floor captures per-encoder scale and bias, and that the Gaussian FID model is valid for pathology features. These are domain assumptions, not established by the paper. No new theoretical entities are introduced; TuroCompress is a proprietary product rather than an invented scientific entity.

free parameters (3)
  • Sample size N = 25,000 per side = 25,000
    All comparisons use N=25,000. The bias in Eq. (2) depends on N and encoder-specific c, so this choice affects every reported distance.
  • k in k-NN precision/recall = not stated
    Eq. (6) defines manifold membership using the k-th nearest neighbor, but the paper never reports the k used for the precision/recall values in Table 6.
  • Per-encoder min-max normalization in perturbation panel
    Eq. (12) rescales each encoder's perturbation sensitivities to [0,1], so the values are comparable only within each encoder, not across encoders.
axioms (5)
  • domain assumption Features of each image set satisfy the Gaussian assumption implicit in the closed-form Fréchet distance (Eq. 1).
    FD is defined between fitted Gaussians; pathology tile features are high-dimensional and may be multimodal, so FD may not capture the true distributional difference.
  • standard math The finite-sample bias expansion E[FD_hat_N] = FD_inf + c/N + o(1/N) (Eq. 2) applies and c depends on the encoder.
    Used to justify sample-size matching; no derivation or verification is given for the pathology feature covariances.
  • ad hoc to paper The within-cohort floor in Eq. (3) is a stable, scale-only baseline and is independent of the comparison being made.
    This is the load-bearing premise of the ratio normalization; the paper does not test floor stability across repeated splits or cohorts.
  • domain assumption The held-out in-house cohort (500k tiles) is a valid primary reference and is representative of clinical pathology tile distributions.
    The cohort is proprietary and not released; all central numbers depend on it.
  • domain assumption Remixed pseudo-slides with a common grid approximate real slide-level composition for the CHIEF encoder.
    In-house tiles lack stored coordinates, so a common grid layout is used; the slide-level result depends on this approximation.

pith-pipeline@v1.3.0-alltime-deepseek · 9227 in / 13918 out tokens · 121066 ms · 2026-08-01T10:50:35.272656+00:00 · methodology

0 comments
read the original abstract

The Frechet Inception Distance (FID) compares two image sets by fitting a Gaussian to the features of a fixed network and measuring the distance between the two Gaussians. In digital pathology the Inception network is routinely replaced by a histology foundation model, on the assumption that a domain encoder gives a more meaningful score. We show that this choice changes the result. For one fixed pair of tile sets, the raw Frechet distance varies about thirty-fold across six common encoders, and the ordering does not follow embedding dimension, so a raw score cannot be read without naming the encoder. Using a held-out in-house cohort (about 500,000 H&E and immunohistochemistry tiles from 2,119 slides) and a public TCGA BRCA cohort (100 slides), we benchmark Inception-v3, Phikon-v2, CONCH, UNI2-h, Virchow2 and Prov-GigaPath across within-cohort baselines, cross-cohort drift, controlled perturbations, compression, stain normalization, and two generative models. Expressing each distance as a ratio to the encoder's own within-cohort floor restores comparability, cutting the across-encoder coefficient of variation by about 89% within cohort and 58% across cohorts. The encoders separate into a sensitive group (CONCH, Phikon-v2, Inception-v3) and an invariant group (UNI2-h, Virchow2, Prov-GigaPath), and this split decides which generative model is judged more realistic, so the encoder can change the conclusion of a generative evaluation. At the slide level, an attention-pooling encoder registers per-slide composition that a pooled patch distance cannot see, raising the distance about 320-fold on matched cohorts. Using the same protocol we evaluate TuroCompress, a proprietary pathology codec, which reaches the highest reconstruction fidelity at the smallest file size among codecs tested. We release the normalization protocol, the per-encoder perturbation panel, and the feature extracts.

Figures

Figures reproduced from arXiv: 2607.20584 by Devansh Lalwani, Jay Mehta, Maulik Shah, Nikita Mulchandani, Rimlee Dutta, Roshani Gala, Sheena Alphones, Swapnil Bhat.

Figure 1
Figure 1. Figure 1: Study workflow: two cohorts, six feature extractors, Fréchet distance with sample-size matching, ratio normalization, and five evaluation settings plus a generative comparison. encoders differ in all three. This paper quantifies how much the reported distance depends on the encoder for a fixed comparison, and shows that the dependence can be removed without discarding the metric. This paper makes three con… view at source ↗
Figure 2
Figure 2. Figure 2: (A) The within-cohort distance floor spans about thirty-fold across encoders for the same com￾parison; bar color marks the two groups identified later. (B) The ratio normalization sharply reduces the across-encoder coefficient of variation on two independent comparisons [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The same two-group ordering of encoders appears for cross-cohort drift, stain shift and generative evaluation. Bars are colored by group [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Representative members of the 25-perturbation panel applied to one tile: geometric, photometric, blur, noise, compression and stain shifts. versus 2.52), whereas under Inception-v3 and Phikon-v2 they were close. A generative-quality conclusion in pathology therefore depends on the encoder used to compute it. Both ratios stayed above one, reflecting a domain gap between either generator’s training distribut… view at source ↗
Figure 5
Figure 5. Figure 5: Two already-compressed H&E tiles (lossless-PNG sizes shown) re-compressed at each codec’s operating point. All preserve diagnostic structure; TuroCompress does so at the smallest size and the highest fidelity. slide is homogeneous (its tiles come from a single tissue cluster); in the other, each slide mixes both clusters equally. Because the pooled tiles match, the patch-level distance stays at the floor (… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 7 linked inside Pith

  1. [1]

    GANs trained by a two time- scale update rule converge to a local Nash equilibrium

    Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S. GANs trained by a two time- scale update rule converge to a local Nash equilibrium. NeurIPS, 2017

    2017

  2. [2]

    The Fréchet distance between multivariate normal distributions

    Dowson DC, Landau BV. The Fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis, 1982;12(3):450–455

    1982

  3. [3]

    Rethinking the Inception architecture for computer vision

    Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the Inception architecture for computer vision. CVPR, 2016, pp. 2818–2826

    2016

  4. [4]

    Effectively unbiased FID and Inception Score and where to find them

    Chong MJ, Forsyth D. Effectively unbiased FID and Inception Score and where to find them. CVPR, 2020. arXiv:1911.07023. 12

    Pith/arXiv arXiv 2020

  5. [5]

    DINOv2: learning robust visual features without supervision

    Oquab M, Darcet T, Moutakanni T, et al. DINOv2: learning robust visual features without supervision. TMLR, 2024. arXiv:2304.07193

    Pith/arXiv arXiv 2024

  6. [6]

    Towards a general-purpose foundation model for computa- tional pathology

    Chen RJ, Ding T, Lu MY, et al. Towards a general-purpose foundation model for computa- tional pathology. Nature Medicine, 2024;30:850–862

    2024

  7. [7]

    A foundation model for clinical-grade computational pathology and rare cancers detection

    Vorontsov E, Bozkurt A, Casson A, et al. A foundation model for clinical-grade computational pathology and rare cancers detection. Nature Medicine, 2024;30:2924–2935

    2024

  8. [8]

    Virchow2: scaling self-supervised mixed magnifi- cation models in pathology

    Zimmermann E, Vorontsov E, Viret J, et al. Virchow2: scaling self-supervised mixed magnifi- cation models in pathology. arXiv:2408.00738, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Scaling self-supervised learning for histopathology with masked image modeling

    Filiot A, Ghermi R, Olivier A, et al. Scaling self-supervised learning for histopathology with masked image modeling. medRxiv 2023.07.21.23292757, 2023

    2023

  10. [10]

    Phikon-v2: a large and public feature extractor for biomarker prediction

    Filiot A, Jacob P, Mac Kain A, Saillard C. Phikon-v2: a large and public feature extractor for biomarker prediction. arXiv:2409.09173, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Nature Medicine, 2024;30:863–874

    LuMY,ChenB,WilliamsonDFK,etal.Avisual-languagefoundationmodelforcomputational pathology. Nature Medicine, 2024;30:863–874

    2024

  12. [12]

    A whole-slide foundation model for digital pathology from real-world data

    Xu H, Usuyama N, Bagga J, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, 2024;630:181–188

    2024

  13. [13]

    Improved precision and recall metric for assessing generative models

    Kynkäänniemi T, Karras T, Laine S, Lehtinen J, Aila T. Improved precision and recall metric for assessing generative models. NeurIPS, 2019. arXiv:1904.06991

    Pith/arXiv arXiv 2019

  14. [14]

    Feature Likelihood Divergence: evaluating the general- ization of generative models using samples

    Jiralerspong M, Bose J, Gemp I, et al. Feature Likelihood Divergence: evaluating the general- ization of generative models using samples. NeurIPS, 2023. arXiv:2302.04440

    Pith/arXiv arXiv 2023

  15. [15]

    Toward a shared vision for cancer genomic data (NCI Genomic Data Commons)

    Grossman RL, Heath AP, Ferretti V, et al. Toward a shared vision for cancer genomic data (NCI Genomic Data Commons). New England Journal of Medicine, 2016;375:1109–1112

    2016

  16. [16]

    PixCell: a generative foundation model for digital histopathology images

    Yellapragada S, Graikos A, Triaridis K, et al. PixCell: a generative foundation model for digital histopathology images. arXiv:2506.05127, 2025

    arXiv 2025

  17. [17]

    SiT: exploring flow and diffusion-based generative models with scalable interpolant transformers

    Ma N, Goldstein M, Albergo MS, Boffi NM, Vanden-Eijnden E, Xie S. SiT: exploring flow and diffusion-based generative models with scalable interpolant transformers. ECCV, 2024. arXiv:2401.08740

    Pith/arXiv arXiv 2024

  18. [18]

    REPA-E: unlocking VAE for end-to-end tuning with latent diffusion transformers

    Leng X, Singh J, Hou Y, Xing Z, Xie S, Zheng L. REPA-E: unlocking VAE for end-to-end tuning with latent diffusion transformers. arXiv:2504.10483, 2025

    arXiv 2025

  19. [19]

    A method for normalizing histology slides for quantitative analysis

    Macenko M, Niethammer M, Marron JS, et al. A method for normalizing histology slides for quantitative analysis. IEEE ISBI, 2009, pp. 1107–1110

    2009

  20. [20]

    ReinhardE,AshikhminM,GoochB,ShirleyP.Colortransferbetweenimages.IEEEComputer Graphics and Applications, 2001;21(5):34–41

    2001

  21. [21]

    Structure-preserving color normalization and sparse stain separation for histological images

    Vahadane A, Peng T, Sethi A, et al. Structure-preserving color normalization and sparse stain separation for histological images. IEEE TMI, 2016;35(8):1962–1971

    2016

  22. [22]

    Image quality assessment: from error visibility to structural similarity

    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004;13(4):600–612. 13

    2004

  23. [23]

    The unreasonable effectiveness of deep features as a perceptual metric

    Zhang R, Isola P, Efros AA, Shechtman E, Wang O. The unreasonable effectiveness of deep features as a perceptual metric. CVPR, 2018, pp. 586–595. 14

    2018