Pith. sign in

REVIEW 2 major objections 3 minor 62 references

Frozen encoders are not blind: a chance-calibrated discordance measure shows they encode clinical findings weakly.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:22 UTC pith:ANAIFMUN

load-bearing objection CANDOR's equal-bank chance calibration is a genuinely useful fix and the empirical map is impressive, but Lemma 1 is overstated and the context-match assumption is load-bearing, so the paper needs revision before the strong claims can stand. the 2 major comments →

arxiv 2607.18451 v1 pith:ANAIFMUN submitted 2026-07-20 cs.LG cs.AIcs.CLcs.CV

CANDOR: Chance-Calibrated Discordance in Frozen Foundation Encoders

Soroosh Tayebi Arasteh , Sven Nebelung , Daniel Truhn This is my paper
classification cs.LG cs.AIcs.CLcs.CV
keywords frozen encodersdiscordance metricchance level calibrationnearest-neighbor geometryfoundation modelsmedical image analysisprevalence biasrepresentation evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Frozen encoders are usually judged by how well a small trained head reads a finding from their features, but a high AUROC can hide the fact that many positive images sit closer to opposite-label images in the encoder's geometry. CANDOR measures this discordance directly and fixes its chance level at exactly one half by using equal-size positive and negative reference banks, a symmetry correction that removes the prevalence confound. Across 22 encoders and 605,443 images, the paper finds that discordance falls below chance in every cell, so no encoder is blind to fine-grained clinical findings, but rates stay high for chest findings and sit at chance for glaucoma and mammographic malignancy, so the encoders are uniformly weak. A Lipschitz margin bound shows that a discordant twin caps the normalized margin any head can achieve on that encoder, yet a per-image selection rule across eleven encoders recovers almost all of the missed cases. The paper's central conclusion is that the deficit is selection, not information: the findings are in the geometry, but no single frozen encoder separates them strongly.

Core claim

The paper's central claim is that the discordance operator D(g;F,s), defined as the share of positive images whose nearest opposite-label neighbors are closer on average than their own kind within a matched acquisition context, has an exact chance level of one half when the two reference banks are equal in size, and that this chance-calibrated measurement reverses the standard reading of frozen encoders. Over 132 encoder-finding cells, every bootstrap interval for the collapse rate sits above 50%, proving no encoder is blind; yet the best chest model, RAD-DINO, still places 18.4% of pneumothorax positives nearer an opposite-label film, and the same encoder that collapses only 4.5% of bird-sp

What carries the argument

The central mechanism is the operator D(g;F,s) = Pr[DM_g(x)>0 | F(x)=1], with DM_g(x) comparing the mean cosine similarity of a positive query to its k nearest neighbors in an equal-size negative bank against the mean to its k nearest neighbors in an equal-size positive bank, both banks drawn from the same acquisition context. The design choice that carries the argument is equal-size banks: they make the two label groups exchangeable under relabeling, so the chance level is fixed at exactly one half by a one-line symmetry argument (Lemma 1) with no empirical null. Behind the operator stands Proposition 1, which converts any discordant twin into a geometric cap on the normalized margin of any

Load-bearing premise

Every discordance rate is interpreted as a property of the finding only if matching images by recorded acquisition covariates (site, projection, and so on) removes all non-finding confounders, and the paper's own ablation pooling across sites and views is inconclusive, so a systematic difference between matched images that correlates with the label—such as an unrecorded scanner protocol or patient habitus—would be misattributed to the finding.

What would settle it

Build a single-site, single-projection context with no real finding difference but with a systematic difference in an unrecorded covariate—for example, label one subset 'positive' and another 'negative' purely by scanner protocol or image brightness. If CANDOR on this pseudo-task returns a collapse rate clearly below 50%, the within-context matching has not removed the confounder and the operator misattributes it to a finding; if it stays at 50%, the matching assumption holds.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If CANDOR is correct, model selection for medical image encoders should be done by measuring geometry directly, not only by head AUROC, since a strong AUROC can coexist with a fifth of positives being placed on the wrong side of the geometry.
  • Because the chance level is fixed, CANDOR can be computed before any head is trained, giving a head-free way to flag which findings a frozen encoder supports poorly.
  • The Lipschitz bound implies that on cases flagged as discordant, increasing head width, depth, or norm cannot increase the normalized margin; the only route to improvement is a different encoder geometry.
  • The near-total recovery of missed cases by per-image selection across the panel implies that ensembling or selection strategies can tap information that no single encoder holds, provided one can solve the label-free selection problem.
  • The association between erasure retention and collapse suggests that encoders that barely move their representation when the evidence is occluded are the ones that place positives beside their opposite, offering a cheap diagnostic for collapse.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The equal-size bank symmetry is a general correction for any nearest-neighbor-based similarity measure with imbalanced groups; published 'blindness' or 'shortcut' results using imbalanced nearest-neighbor statistics may need to be re-read as prevalence artifacts.
  • The per-image selection result suggests a practical routing system that picks an encoder per image, but the paper shows a label-free proxy for 'which encoder to trust' remains open; an unsupervised version of that selector would be a direct extension.
  • If erasure retention is causally related to collapse, a training objective that penalizes retention when evidence is removed might directly reduce discordance; this is a testable hypothesis the paper does not pursue.
  • The finding that general-purpose encoders collapse less on natural fine-grained tasks (birds, aircraft) than on chest findings suggests that the weakness is task-specific, possibly tied to the subtlety and low-level nature of medical findings; this could guide the design of medical-specific encoders.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper introduces CANDOR, a nearest-neighbor discordance operator D(g;F,s)=Pr[DM_g(x)>0 | F(x)=1] built from equal-size positive/negative reference banks matched on acquisition context (site-by-view for chest, source dataset elsewhere). It claims an exact chance level of 1/2 under the null, proves a Lipschitz margin bound on any head, and reports a large empirical study (22 encoders, 20 datasets, 605,443 images) in which collapse rates fall below 50% almost everywhere, leading to the conclusion that frozen encoders encode fine-grained findings weakly but are not blind. The paper also analyzes blind-set overlap across encoders, correlates erasure retention with collapse, and shows that an oracle per-image selector among 11 heads achieves 2.8% error where a single head misses 35.9%.

Significance. If correct, CANDOR provides a head-free, chance-calibrated measurement for evaluating frozen encoders, potentially before training any readout. The study's strengths include a careful empirical design: paired hashed rows shared across encoders, bootstrap CIs, FDR correction, synthetic planted-encoder calibration, and a released code repository. The Lipschitz margin bound is a useful theoretical contribution, as is the demonstration that several standard uncertainty scores collapse into one detector. However, the central chance-level theorem as stated is imprecise, and the interpretation of the rates as finding-specific rests on an explicitly unverifiable no-confounder assumption.

major comments (2)
  1. [CANDOR Framework, Lemma 1 ('The chance level is exactly one half')] Lemma 1 states that for any encoder g, Pr[DM_g(x)>0]=1/2 when B+ and B- are drawn as two equal-sized blocks uniformly from the context. This is not the sampling distribution of the operator in Eq. (2), where B+ and B- are label-conditioned banks. Exchangeability of the two blocks holds only under H0 (g(X)⊥F|s) or under a permutation null, not for an arbitrary informative encoder. As written, the lemma is contradicted by the planted 'sees it' encoder scoring 0.0 (Ablation 1, Fig. 2a) and by every below-50 rate in Table 2. Please restate the claim as 'under H0, the expected discordance is 1/2' and make the probability space explicit. This is the central chance-calibration result and needs correction.
  2. [CANDOR Framework, Problem formulation; Ablation 4] The interpretation of a CANDOR rate as measuring whether an encoder separates a specific finding assumes that matching within an acquisition context removes all non-finding confounders. The paper itself calls this 'the one assumption the data cannot check.' Ablation 4 reports that pooling banks across sites/views is inconclusive: the matched rate is the lower one in only 9 of 22 cells. Unrecorded covariates (severity, patient habitus, image processing, co-occurring devices) could therefore drive the discordance and produce the 'weak encoding' conclusion even if the encoder is not encoding the finding. The headline claims in the abstract and conclusion should be qualified as measuring label separation within the recorded context, or supplemented with analyses that directly address specific plausible confounders.
minor comments (3)
  1. [Throughout] The manuscript contains many missing spaces and odd capitalizations (e.g., 'Chance-cAlibrated NeighborhoodDiscORdance'), likely from a rendering artifact. Please ensure a cleanly formatted manuscript, especially in the abstract and throughout the text.
  2. [Table 4] The column labeled 'Best' is the per-image oracle over the 11 heads. Please clarify this in the caption, since the reader might otherwise interpret it as a single trained head.
  3. [Ablation 1 / Figure 2a] The planted-blind and planted-sees encoders are described with inconsistent hyphenation ('planted blind' vs 'planted-blind'). Please standardize.

Circularity Check

0 steps flagged

No significant circularity: the chance level is analytic and the empirical collapse rates are not fitted to the conclusion.

full rationale

The derivation chain is self-contained at the points that matter. Lemma 1 fixes the chance level at exactly one half by a symmetry argument on equal-size banks; no empirical null is fitted, and the planted-blind experiment (49.5, CI covering 50) validates the estimator rather than calibrating any constant used in the reported rates. The below-chance collapse rates are computed directly from the fixed operator D(g;F,s), with no parameter fit to produce the 'weak but not blind' conclusion. Proposition 1 is an analytic Lipschitz-margin inequality, not a fitted result. The paper's own flagged limitation is the context-match assumption ('only as good as the covariates recorded, the one assumption the data cannot check') and the inconclusive Ablation 4; that is a residual confounding / external-validity concern, not a circular reduction. The erasure-retention Spearman 0.855 is explicitly correlational ('stays correlational and establishes no mechanism on its own') and is convergent evidence, not a fitted prediction. The oracle 'some head among eleven' comparison is openly label-dependent ('shown the label and picking, per image, whichever head is right') and is presented as an upper bound, not as a deployable prediction. Self-citations appear only in motivational or contextual roles (e.g., resolution governing transfer, VLM behavior, panel coordination) and none is load-bearing for the chance-level or collapse claims. No step reduces Eq. (1)/(2) to its own inputs, and no fitted parameter is renamed as a prediction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical entities are postulated. The 'discordant twin' is a defined neighborhood condition, not a new force or object; the planted blind/sees-it encoders are experimental constructs used to calibrate the estimator. Free parameters are mostly benign hyperparameters; the context partition is the only one that materially shapes interpretation.

free parameters (4)
  • k = 5 nearest neighbors
    Hand-chosen neighborhood size; the chance-level claim (Lemma 1) is independent of k, so this does not manufacture the result, but it sets the scale of the reported rates.
  • Reference bank cap of 4,000 rows per equalized bank
    Compute-driven subsampling cap; the paper notes rates on large sites are subsample estimates, not censuses. The chance level is unaffected, but precision on large sites is reduced.
  • Context partition (site-by-view for chest; source dataset elsewhere; coarse superclass for natural images)
    The load-bearing modeling choice behind 'only the finding differs'; the paper calls the sufficiency of recorded covariates 'the one assumption the data cannot check'.
  • Head/readout hyperparameters (30 epochs, Adam 1e-3, weight decay 1e-4, hidden 256, sensitivity-80 operating point)
    Standard training choices for the linear/256-unit heads; they affect AUROC and head-error numbers but not the CANDOR operator itself.
axioms (4)
  • domain assumption Null H0: g(X) independent of F given context s (features carry no within-context information about the finding)
    The exact 1/2 chance level is derived under this null ('Under H0 ... the expected discordance is exactly 1/2'); Lemma 1 as headline-stated drops this condition and becomes false.
  • standard math Induced similarities are almost surely free of ties
    Required for P[DM>0] = P[DM<0] with no atom at 0; benign for continuous features, but banks are capped at 4,000 rows and k=5, so ties are not impossible.
  • domain assumption Public dataset labels are correct enough for the bank definitions
    Label noise affects both banks; partially addressed by the box-verified MS-CXR/RSNA ablations, which the paper reports as consistent with the headline rates.
  • domain assumption Heads are L-Lipschitz on the unit sphere
    Standard for the certified-robustness inequality imported from [20]; needed for Proposition 1's normalized-margin cap.

pith-pipeline@v1.3.0-alltime-deepseek · 15050 in / 26887 out tokens · 223824 ms · 2026-08-01T15:22:39.145355+00:00 · methodology

0 comments
read the original abstract

Frozen encoders are chosen by how well a lightweight head reads a finding from their features, not whether the geometry separates it. Nearest-neighbor discordance does, but with unequal banks the opposite-label neighbor wins on density, not geometry, so prevalence alone makes an uninformed encoder look blind. We introduce CANDOR, a discordance measure whose equal-size banks are symmetric under a label swap, fixing its chance level at exactly one half. Across 22 encoders, 20 datasets from 7 domains, and 605,443 images, this correction reverses the conclusion. Collapse falls below chance almost everywhere, so no encoder is blind, yet all are weak: the best chest model reads pneumothorax at 84.5 AUROC and still places 18.4% of those positives nearer an opposite-label film than its own kind in the same hospital. The same encoder that resolves bird species at 4.5 leaves chest findings at 42.8 and glaucoma at 49.8, at chance and worse than random weights. Such a case caps the normalized margin of any Lipschitz head, yet some head among eleven is correct on all but 2.8% of cases where one head misses 35.9%: the deficit is selection, not information. Erasure retention is associated with collapse; we detect no association with the objective, scale, recency, or size of the finding. Because the chance level is fixed, CANDOR can be read before any head is trained, flagging which findings a frozen encoder supports poorly.

Figures

Figures reproduced from arXiv: 2607.18451 by Daniel Truhn, Soroosh Tayebi Arasteh, Sven Nebelung.

Figure 1
Figure 1. Figure 1: A real pneumothorax film (finding outlined) and its five nearest RAD-DINO neighbors, same site and view, all [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) The two planted encoders bracket the eleven chest encoders.(b)Collapse per task, one marker per encoder. Irreducible per encoder but recoverable across the panel Proposition 1 says a discordant twin caps the normalized margin of any Lipschitz head on that encoder. On a synthetic sweep of class overlap the achieved joint margin never ex￾ceeds the bound on any of 5,000 test pairs, while the bound itself … view at source ↗
Figure 3
Figure 3. Figure 3: (a) The chest encoders ordered by erasure retention, with collapse on the same axis. (b) Retention on IDRiD. A label-free flag does not beat confidence The neighborhood impurity score r of Eq. 5 is the natural label-free candidate: same geometry, no query label, one forward pass. It does not work. Against the head’s own errors it reaches 52.2 AUROC on chest, pooled over six sites and 573 cells, and barely … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 1 canonical work pages

  1. [1]

    Enhancing diagnosticdeeplearningviaself-supervisedpretraining on large-scale, unlabeled non-medical images.Euro- pean Radiology Experimental, 8(1):10, 2024

    Soroosh Tayebi Arasteh, Leo Misera, Jakob Nikolas Kather, Daniel Truhn, and Sven Nebelung. Enhancing diagnosticdeeplearningviaself-supervisedpretraining on large-scale, unlabeled non-medical images.Euro- pean Radiology Experimental, 8(1):10, 2024

    2024

  2. [2]

    Mahshad Lotfinia, Arash Tayebiarasteh, Samaneh Samiei, Mehdi Joodaki, and Soroosh Tayebi Arasteh. Boosting multi-demographic federated learning for chest radiograph analysis using general-purpose self- supervisedrepresentations.EuropeanJournalofRadi- ology Artificial Intelligence, 3:100028, 2025

    2025

  3. [3]

    Understanding intermediatelayersusinglinearclassifierprobes,2018

    Guillaume Alain and Yoshua Bengio. Understanding intermediatelayersusinglinearclassifierprobes,2018. URL https://arxiv.org/abs/1610.01644

    Pith/arXiv arXiv 2018

  4. [4]

    Eyes Wide Shut? Ex- ploringtheVisualShortcomingsofMultimodalLLMs

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes Wide Shut? Ex- ploringtheVisualShortcomingsofMultimodalLLMs . InCVPR 2024, pages 9568–9578, June 2024. doi: 10.1109/CVPR52733.2024.00914

    arXiv 2024

  5. [5]

    Frozen foundation-model embeddings discard small- lesionsignalinchestradiography:Implicationsforpre- deployment evaluation, 2026

    Raajitha Muthyala, Zhenan Yin, Alekhya Jilla, Frank Li,TheoDapamede,BardiaKhosravi,Mohammadreza Chavoshi, Judy Gichoya, and Saptarshi Purkayastha. Frozen foundation-model embeddings discard small- lesionsignalinchestradiography:Implicationsforpre- deployment evaluation, 2026. URL https://arxiv.org/ abs/2606.11606

    Pith/arXiv arXiv 2026

  6. [6]

    Vision-language models for chest radiog- raphy do not always need the image, 2026

    MahshadLotfinia,SebastianZiegelmayer,LisaAdams, Daniel Truhn, Andreas Maier, and Soroosh Tayebi Arasteh. Vision-language models for chest radiog- raphy do not always need the image, 2026. URL https://arxiv.org/abs/2606.17710

    Pith/arXiv arXiv 2026

  7. [7]

    Preva- lence of neural collapse during the terminal phase of deep learning training.Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020

    VardanPapyan,XYHan,andDavidLDonoho. Preva- lence of neural collapse during the terminal phase of deep learning training.Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020

    2020

  8. [8]

    A baseline for detecting misclassified and out-of-distribution exam- ples in neural networks

    Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution exam- ples in neural networks. InICLR, 2017. URL https: //openreview.net/forum?id=Hkg4TI9xl

    2017

  9. [9]

    Energy-based out-of-distribution detection

    WeitangLiu,XiaoyunWang,JohnD.Owens,andYix- uan Li. Energy-based out-of-distribution detection. In NeurIPS 2020, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546

    2020

  10. [10]

    A simple unified framework for detecting out- of-distribution samples and adversarial attacks

    Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. In NeurIPS 2018, NIPS’18, page 7167–7177, 2018

    2018

  11. [11]

    Simple and scalable predictive un- certaintyestimationusingdeepensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive un- certaintyestimationusingdeepensembles. InNeurIPS 2017,NIPS’17,page6405–6416,RedHook,NY,USA, 2017

    2017

  12. [12]

    Out-of-distribution detection with deep nearest neigh- bors

    Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neigh- bors. InICLR2022,pages20827–20840.PMLR,2022

    2022

  13. [13]

    Selective clas- sification for deep neural networks

    Yonatan Geifman and Ran El-Yaniv. Selective clas- sification for deep neural networks. InNeurIPS 2017, NIPS’17,page4885–4894,RedHook,NY,USA,2017

    2017

  14. [14]

    Position: the platonic representation hy- pothesis

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: the platonic representation hy- pothesis. InICML 2024, ICML’24, 2024

    2024

  15. [15]

    Mat- szangosz, Gergely Papp, and Dániel Varga

    Adrián Csiszárik, Péter Kőrösi-Szabó, Ákos K. Mat- szangosz, Gergely Papp, and Dániel Varga. Similar- ity and matching of neural network representations. InNeurIPS 2021, 2021. URL https://openreview.net/ forum?id=aedFIIRRfXr

    2021

  16. [16]

    Shortcut learning in deep neural networks.Nature Machine Intelligence, 2 (11):665–673, 2020

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2 (11):665–673, 2020

    2020

  17. [17]

    Ai for radiographic covid-19 detectionselectsshortcutsoversignal.NatureMachine Intelligence, 3(7):610–619, 2021

    Alex J DeGrave et al. Ai for radiographic covid-19 detectionselectsshortcutsoversignal.NatureMachine Intelligence, 3(7):610–619, 2021

    2021

  18. [18]

    Agentic retrieval-augmented reasoning reshapes col- lective reliability under model variability in radiology question answering, 2026

    Mina Farajiamiri, Jeta Sopa, Saba Afza, Lisa Adams, Felix Barajas Ordonez, Tri-Thien Nguyen, Mahshad Lotfinia, Sebastian Wind, Keno Bressem, Sven Nebelung, Daniel Truhn, and Soroosh Tayebi Arasteh. Agentic retrieval-augmented reasoning reshapes col- lective reliability under model variability in radiology question answering, 2026. URL https://arxiv.org/ab...

    arXiv 2026

  19. [19]

    The strength of clinical ev- idence is recoverable from language model represen- tations but not from their stated grades, 2026

    Soroosh Tayebi Arasteh. The strength of clinical ev- idence is recoverable from language model represen- tations but not from their stated grades, 2026. URL https://arxiv.org/abs/2606.29034

    Pith/arXiv arXiv 2026

  20. [20]

    Lipschitz-margintraining:Scalablecertificationofper- turbationinvariancefordeepneuralnetworks.NeurIPS 2018, 31, 2018

    Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margintraining:Scalablecertificationofper- turbationinvariancefordeepneuralnetworks.NeurIPS 2018, 31, 2018

    2018

  21. [21]

    Resolution scaling governs dinov3 transferperformanceinchestradiographclassification,

    Soroosh Tayebi Arasteh, Mina Shaigan, Christiane Kuhl, Jakob Nikolas Kather, Sven Nebelung, and Daniel Truhn. Resolution scaling governs dinov3 transferperformanceinchestradiographclassification,

  22. [22]

    Mimic- cxr, a de-identified publicly available database of chest radiographswithfree-textreports.Scientificdata,6(1): 317, 2019

    AlistairEWJohnson,TomJPollard,SethJBerkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih- ying Deng, Roger G Mark, and Steven Horng. Mimic- cxr, a de-identified publicly available database of chest radiographswithfree-textreports.Scientificdata,6(1): 317, 2019

    2019

  23. [23]

    Chexpert: A large chest radiograph dataset with un- certainty labels and expert comparison

    Jeremy Irvin, Pranav Rajpurkar, Michael Ko, et al. Chexpert: A large chest radiograph dataset with un- certainty labels and expert comparison. InProceed- ings of the AAAI conference on artificial intelligence, volume 33, pages 590–597, 2019

    2019

  24. [24]

    Padchest: A large chest x-ray image dataset with multi-label annotated re- ports.MedicalImageAnalysis,66:101797,2020

    Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia-Vayá. Padchest: A large chest x-ray image dataset with multi-label annotated re- ports.MedicalImageAnalysis,66:101797,2020. ISSN 1361-8415. doi: https://doi.org/10.1016/j.media.2020. 101797

    work page doi:10.1016/j.media.2020 2020

  25. [25]

    Vindr- cxr: An open dataset of chest x-rays with radiologist’s annotations.Scientific Data, 9(1):429, 2022

    Ha Q Nguyen, Khanh Lam, Linh T Le, et al. Vindr- cxr: An open dataset of chest x-rays with radiologist’s annotations.Scientific Data, 9(1):429, 2022

    2022

  26. [26]

    Hieu H Pham, Ngoc H Nguyen, Thanh T Tran, TuanNMNguyen,andHaQNguyen.Pedicxr:anopen, large-scalechestradiographdatasetforinterpretationof common thoracic diseases in children.Scientific data, 10(1):240, 2023

    2023

  27. [27]

    Preparing a collection of radiology examinations for distribution and retrieval.Journal of the Ameri- can Medical Informatics Association, 23(2):304–310, 2016

    DinaDemner-Fushman,MarcDKohli,MarcBRosen- man, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDon- ald. Preparing a collection of radiology examinations for distribution and retrieval.Journal of the Ameri- can Medical Informatics Association, 23(2):304–310, 2016

    2016

  28. [28]

    Augmenting the na- tional institutes of health chest radiograph dataset with expert annotations of possible pneumonia.Radiology: Artificial Intelligence, 1(1):e180041, 2019

    George Shih, Carol C Wu, et al. Augmenting the na- tional institutes of health chest radiograph dataset with expert annotations of possible pneumonia.Radiology: Artificial Intelligence, 1(1):e180041, 2019

    2019

  29. [29]

    Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, Hoifung Poon, and Ozan Oktay

    Benedikt Boecking, Naoto Usuyama, Shruthi Ban- nur, Daniel C. Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, Hoifung Poon, and Ozan Oktay. Making the most of text semantics to improve biomedical vision–language processing. InECCV 2022, page 1–21, 2022. doi: 10.1007/978-3-031- 20059-5_1

    doi:10.1007/978-3-031- 2022

  30. [30]

    100,000histologicalimagesofhumancolorectal cancer and healthy tissue

    Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 100,000histologicalimagesofhumancolorectal cancer and healthy tissue. 2018

    2018

  31. [31]

    Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling

    Bastiaan S. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. InMICCAI 2018, page 210–218, 2018. doi: 10.1007/978-3-030-00934-2_24

    doi:10.1007/978-3-030-00934-2_24 2018

  32. [32]

    The caltech-ucsd birds- 200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds- 200-2011 dataset. 2011

    2011

  33. [33]

    Fine-grained visual classification of aircraft, 2013

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft, 2013. URL https://arxiv.org/ abs/1306.5151

    Pith/arXiv arXiv 2013

  34. [34]

    Fairvision: Equitable deep learning for eye dis- ease screening via fair identity scaling, 2024

    Yan Luo, Muhammad Osama Khan, Yu Tian, Min Shi, Zehao Dou, Tobias Elze, Yi Fang, and Mengyu Wang. Fairvision: Equitable deep learning for eye dis- ease screening via fair identity scaling, 2024. URL https://arxiv.org/abs/2310.02492

    Pith/arXiv arXiv 2024

  35. [35]

    Aptos 2019 blind- ness detection

    Maggie Karthik and Sohier Dane. Aptos 2019 blind- ness detection. kaggle, 2019

    2019

  36. [36]

    Feedbackonapubliclydistributedimagedatabase:the messidordatabase.ImageAnalysis&Stereology,pages 231–234, 2014

    EtienneDecencière,XiweiZhang,GuyCazuguel,etal. Feedbackonapubliclydistributedimagedatabase:the messidordatabase.ImageAnalysis&Stereology,pages 231–234, 2014

    2014

  37. [37]

    Indiandiabeticretinopa- thy image dataset (idrid): a database for diabetic retinopathy screening research.Data, 3(3):25, 2018

    Prasanna Porwal, Samiksha Pachade, Ravi Kamble, ManeshKokare,GirishDeshmukh,VivekSahasrabud- dhe,andFabriceMeriaudeau. Indiandiabeticretinopa- thy image dataset (idrid): a database for diabetic retinopathy screening research.Data, 3(3):25, 2018

    2018

  38. [38]

    Evaluating deep neural networks trained on clinicalimagesindermatologywiththefitzpatrick17k dataset

    MatthewGroh,CalebHarris,LuisSoenksen,FelixLau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinicalimagesindermatologywiththefitzpatrick17k dataset. InCVPR 2021, pages 1820–1828, 2021

    2021

  39. [39]

    Bcn20000: Dermoscopic lesions in the wild.Scien- tific data, 11(1):641, 2024

    Carlos Hernández-Pérez, Marc Combalia, et al. Bcn20000: Dermoscopic lesions in the wild.Scien- tific data, 11(1):641, 2024

    2024

  40. [40]

    Novoa,etal.Disparitiesindermatologyaiperformance on a diverse, curated clinical image set.Science Ad- vances, 8(32):eabq6147, 2022

    Roxana Daneshjou, Kailas Vodrahalli, Roberto A. Novoa,etal.Disparitiesindermatologyaiperformance on a diverse, curated clinical image set.Science Ad- vances, 8(32):eabq6147, 2022. doi: 10.1126/sciadv. abq6147

    doi:10.1126/sciadv 2022

  41. [41]

    Vindr-mammo: A large-scale benchmark dataset for computer-aideddiagnosisinfull-fielddigitalmammog- raphy.Scientific Data, 10(1):277, 2023

    Hieu T Nguyen, Ha Q Nguyen, Hieu H Pham, Khanh Lam, Linh T Le, Minh Dao, and Van Vu. Vindr-mammo: A large-scale benchmark dataset for computer-aideddiagnosisinfull-fielddigitalmammog- raphy.Scientific Data, 10(1):277, 2023

    2023

  42. [42]

    Exploring scalable medical image encoders beyond text supervi- sion.NatureMachineIntelligence,7(1):119–130,2025

    Fernando Pérez-García, Harshita Sharma, Sam Bond- Taylor, Kenza Bouzid, Valentina Salvatelli, Maxim- ilian Ilse, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Matthew P Lungren, et al. Exploring scalable medical image encoders beyond text supervi- sion.NatureMachineIntelligence,7(1):119–130,2025

    2025

  43. [43]

    Amultimodal biomedical foundation model trained from fifteen mil- lion image–text pairs.Nejm Ai, 2(1):AIoa2400640, 2025

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Ra- jeshRao,MuWei,NaveenValluri,etal. Amultimodal biomedical foundation model trained from fifteen mil- lion image–text pairs.Nejm Ai, 2(1):AIoa2400640, 2025

    2025

  44. [44]

    Torchxrayvision: A library of chest x-ray datasets and models

    JosephPaulCohen,JosephDViviano,PaulBertin,Paul Morrison,ParsaTorabian,MatteoGuarrera,MatthewP Lungren, Akshay Chaudhari, Rupert Brooks, Moham- mad Hashir, et al. Torchxrayvision: A library of chest x-ray datasets and models. InMIDL, pages 231–249. PMLR, 2022

    2022

  45. [45]

    Phikon-v2, a large and public fea- ture extractor for biomarker prediction, 2024

    Alexandre Filiot, Paul Jacob, Alice Mac Kain, and Charlie Saillard. Phikon-v2, a large and public fea- ture extractor for biomarker prediction, 2024. URL https://arxiv.org/abs/2409.09173

    Pith/arXiv arXiv 2024

  46. [46]

    Towards a general-purpose foun- dation model for computational pathology.Nature medicine, 30(3):850–862, 2024

    Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muham- mad Shaban, et al. Towards a general-purpose foun- dation model for computational pathology.Nature medicine, 30(3):850–862, 2024

    2024

  47. [47]

    Nature medicine, 30(10):2924–2935, 2024

    Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Kristen Sev- erson,EricZimmermann,JamesHall,NeilTenenholtz, NicoloFusi,etal.Afoundationmodelforclinical-grade computational pathology and rare cancers detection. Nature medicine, 30(10):2924–2935, 2024

    2024

  48. [48]

    Virchow2: Scaling self-supervised mixed magnification models in pathology, 2024

    EricZimmermann,EugeneVorontsov,etal. Virchow2: Scaling self-supervised mixed magnification models in pathology, 2024. URL https://arxiv.org/abs/2408. 00738

    2024

  49. [49]

    A whole-slide foundationmodelfordigitalpathologyfromreal-world data.Nature, 630(8015):181–188, 2024

    Hanwen Xu, Naoto Usuyama, et al. A whole-slide foundationmodelfordigitalpathologyfromreal-world data.Nature, 630(8015):181–188, 2024

    2024

  50. [50]

    Avisual-languagefoundationmodelforcomputational pathology.Nature medicine, 30(3):863–874, 2024

    Ming Y Lu, Bowen Chen, Drew FK Williamson, et al. Avisual-languagefoundationmodelforcomputational pathology.Nature medicine, 30(3):863–874, 2024

    2024

  51. [51]

    Afoundationmodelforgeneralizablediseasedetection fromretinalimages.Nature,622(7981):156–163,2023

    Yukun Zhou, Mark A Chia, Siegfried K Wagner, et al. Afoundationmodelforgeneralizablediseasedetection fromretinalimages.Nature,622(7981):156–163,2023

    2023

  52. [52]

    A foundation language- image model of the retina (flair): Encoding expert knowledge in text supervision.Medical Image Anal- ysis, 99:103357, 2025

    JulioSilva-Rodriguez,HadiChakor,RiadhKobbi,Jose Dolz, and Ismail Ben Ayed. A foundation language- image model of the retina (flair): Encoding expert knowledge in text supervision.Medical Image Anal- ysis, 99:103357, 2025

    2025

  53. [53]

    Transparent medical image ai via an image–text foundation model grounded in medical lit- erature.Nature medicine, 30(4):1154–1165, 2024

    Chanwoo Kim, Soham U Gadgil, Alex J DeGrave, Je- sutofunmiAOmiye,ZhuoRanCai,RoxanaDaneshjou, and Su-In Lee. Transparent medical image ai via an image–text foundation model grounded in medical lit- erature.Nature medicine, 30(4):1154–1165, 2024

    2024

  54. [54]

    A multi- modal vision foundation model for clinical dermatol- ogy.Nature Medicine, 31(8):2691–2702, 2025

    Siyuan Yan, Zhen Yu, Clare Primiero, et al. A multi- modal vision foundation model for clinical dermatol- ogy.Nature Medicine, 31(8):2691–2702, 2025

    2025

  55. [55]

    Vo, Maximilian Seitzer, et al

    Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, et al. Dinov3, 2025. URL https://arxiv.org/abs/2508.10104

    Pith/arXiv arXiv 2025

  56. [56]

    DINOv2: Learning robust visual features with- out supervision.Transactions on Machine Learn- ing Research, 2024

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, et al. DINOv2: Learning robust visual features with- out supervision.Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https: //openreview.net/forum?id=a68SUt6zFt

    2024

  57. [57]

    Learningtransferablevisualmodels from natural language supervision

    AlecRadfordetal. Learningtransferablevisualmodels from natural language supervision. InICLR, pages 8748–8763. PmLR, 2021

    2021

  58. [58]

    Siglip2:Mul- tilingual vision-language encoders with improved se- manticunderstanding,localization,anddensefeatures,

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, YeXia,BasilMustafa,OlivierHénaff,JeremiahHarm- sen,AndreasSteiner,andXiaohuaZhai. Siglip2:Mul- tilingual vision-language encoders with improved se- manticunderstanding,localization,anddensefeatures,

  59. [59]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv. org/abs/1412.6980

    Pith/arXiv arXiv 2017

  60. [60]

    Controlling the false discovery rate: a practical and powerful approach to multiple testing.Journal of the Royal statistical so- ciety:seriesB(Methodological),57(1):289–300,1995

    Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing.Journal of the Royal statistical so- ciety:seriesB(Methodological),57(1):289–300,1995

    1995

  61. [2025]

    URL https://arxiv.org/abs/2502.14786

    Pith/arXiv arXiv

  62. [2026]

    URL https://arxiv.org/abs/2510.07191

    Pith/arXiv arXiv