REVIEW 3 major objections 5 minor 20 references
Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A frozen 3D CT encoder's ability to detect a finding is set by the finding's contrast and spatial extent, not by the model's architecture.
desk verdict A careful, reproducible benchmark whose empirical difficulty-ordering result is credible and useful; the global-pooling causal story is a plausible inference, not a demonstrated mechanism. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the global pooled embedding: a single vector formed from the entire scan volume, typically by aggregating patch or token features, which is what all three readouts consume. The analysis that isolates the paper's physical claim is the set of seven within-organ comparison pairs, two findings in the same organ matched to differ mainly in contrast or mainly in extent, measured by macro AUROC averaged over ten encoders. The readout decomposition (cosine k-NN, zero-shot text prompting, linear probing) shows that the difficulty hierarchy is not an artifact of a weak readout, because the linear probe extracts more signal yet still fails on focal lesions.
What would settle it
Measure the actual Hounsfield-unit contrast and volumetric extent of every finding in the seven comparison pairs and check whether AUROC differences track the measured values; alternatively, insert synthetic lesions of fixed contrast and varying size into scans and recover them with a frozen encoder. If small bright lesions are recovered as easily as large bright ones, the extent axis of the claim fails; if large low-contrast lesions are missed, the contrast axis fails.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that a finding's detectability in a frozen 3D CT encoder is governed by two physical properties, contrast against surrounding tissue and spatial extent, rather than by model architecture or scale. This is shown with controlled within-organ comparisons: holding extent fixed while raising contrast (effusion versus thickening, solid versus subsolid nodule) improves macro AUROC by about +0.18, and holding contrast fixed while increasing extent (bulk versus faint calcification, diffuse versus focal ground-glass) improves it by about +0.16. All seven pairs show the same direction of effect with bootstrap significance, and 68 of 70 encoder-pair comparisons agree. The same easy-to-hard ordering appears across an unseen internal cohort and across k-NN, zero-shot, and linear-probe readouts, so the paper concludes that globally pooled embeddings inherently attenuate the small spatial signals of focal lesions and that those lesions will remain unrecoverable unless future pretraining uses region- or lesion-level objectives.
Load-bearing premise
The load-bearing premise is that the seven hand-picked within-organ comparison pairs isolate contrast or extent from everything else; those pairs also differ in prevalence, label-extraction pipeline, and radiological definition, so if this isolation fails, the physical-bottleneck claim reduces to saying conspicuous findings are conspicuous.
Editorial extensions
If this is right
- If the claim holds, clinicians should not rely on frozen global CT embeddings for small low-contrast incidental findings; those findings need spatially grounded detection.
- Benchmarking should report per-finding-type results rather than a single macro AUROC, because global scores hide the physical ceiling.
- Pretraining objectives that operate on regions or lesions, rather than whole volumes, are the direct route to closing the focal-lesion gap.
- Fine-grained tokenization and vision-language alignment improve retrieval and zero-shot ability but do not remove the physical bottleneck; explicit labels can substitute for scale on easy findings but not on small low-contrast ones.
- For rare findings, the small zero-shot-over-kNN AUROC advantage vanishes under prevalence-normalized skill, so readout choice matters mainly for common abnormalities.
Reading between the lines
- The same physical ceiling should appear in any volumetric medical-imaging encoder that pools globally, including MRI and PET, so a direct test in those modalities would show whether the mechanism generalizes.
- One testable extension is to pretrain with a contrastive loss that operates on local patches or detected regions and compare against a global-pooling baseline on small lesions; the paper's account predicts the local objective should break the ceiling.
- The consistency across three independently labeled cohorts suggests label noise is not the driver, but a causal version of the claim would need measured Hounsfield-unit contrast and lesion volume rather than qualitative assignment for each comparison pair.
- The near-universal directionality across encoders implies that gain from better architecture concentrates on easy findings, so aggregate leaderboards may mislead model selection for radiology workflows.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks ten frozen 3D CT encoders on three thoracic CT datasets (CT-RATE, RAD-ChestCT, and an unseen internal hospital cohort) using three readouts (kNN, zero-shot prompting, and linear probing), reporting AUROC and prevalence-normalized skill. It finds no universal state-of-the-art, with fine-grained tokenization plus vision–language alignment generally leading but a lightweight supervised encoder remaining competitive. The central empirical observation is a consistent difficulty ordering across encoders and cohorts (Kendall W = 0.89): devices and fluid are easy, focal low-contrast lesions are hard. The authors attribute this to a physical bottleneck (contrast and spatial extent) and conclude that globally pooled embeddings cannot adequately represent small, low-contrast structures, recommending region- or lesion-level pretraining.
Significance. The benchmark is valuable in several respects: it evaluates a truly unseen clinical cohort, uses patient-grouped cross-validation and bootstrap confidence intervals, covers three distinct readouts, and provides code for the public cohorts. The consistent difficulty hierarchy across ten encoders and three cohorts is a robust empirical contribution that can inform model selection and evaluation design, independent of whether the causal explanation is accepted. The paper also gives credit to the possibility that explicit labels can substitute for scale, a finding that is interesting for practitioners. The weakness is the gap between the observed correlation and the causal claim about global pooling, which is not directly tested. If the causal claim is confirmed, the paper would have significant implications for pretraining objectives.
major comments (3)
- [Section 4, Finding Types; Fig. 4] The within-organ comparisons are presented as isolating contrast and spatial extent, but the pairs differ in multiple other dimensions beyond the claimed physical axis. Contrast and extent are assigned qualitatively, and the pairs also differ in prevalence, label-extraction pipeline, and radiological definition (e.g., pleural effusion vs. thickening; solid vs. subsolid nodule). The bootstrap-significant AUROC differences and the 68/70 directionality agreement establish a highly consistent ordering, but not that the intended physical properties are the cause. To support the causal claim, the authors should quantify contrast (e.g., HU difference to surrounding tissue) and extent (e.g., lesion volume or segmentation) for each pair, and provide a per-pair table reporting prevalence and label source, so that these potential confounds can be assessed or adjusted.
- [Section 3, Discussion; Abstract] The central attribution to "globally pooled embeddings" is not directly tested. No experiment compares these frozen encoders against region-level or lesion-level features under the same protocol, and no ablation removes pooling (e.g., by using patch embeddings from the same encoders instead of the global vector). The cited spatial-localization systems (Sybil, LungEvaty) are not evaluated here. As written, the conclusion is a hypothesis consistent with the data rather than a demonstrated mechanism. The authors should either add a comparison or ablation that directly addresses pooling, or rephrase the conclusion as a testable prediction, noting the absence of direct evidence.
- [Section 4, Datasets; Discussion, Limitations] The label-noise concern is acknowledged but dismissed based on the consistency of the difficulty ordering across independently constructed pipelines (Kendall W = 0.89). This does not rule out the confound: all three pipelines derive labels from radiology reports, whose language is systematically less reliable for subtle findings (subsolid nodules, ground-glass opacities), so the shared report-language bias could produce the same ordering even if the embeddings were insensitive to focal lesions. Provide a control using adjudicated labels or label-confidence scores (e.g., from the NLP extractors) to show that the ordering persists when label noise is reduced, or explicitly account for label uncertainty in the analysis.
minor comments (5)
- [Section 4, Models] The term "globally pooled embeddings" is used for all ten models, but a few models use graph/CNN pooling or other aggregation schemes. Specify the actual pooling mechanism for each encoder, or restrict the claim to models with global average pooling.
- [Fig. 5] Figure 5 is very dense and nearly illegible at column width; consider splitting it per cohort or enlarging the font.
- [Throughout] The dataset name is inconsistently written as "RadChestCT", "RAD-Chest CT", and "RAD-ChestCT"; unify the naming.
- [Section 2, Results] Kendall's W is reported without a confidence interval or p-value; report a bootstrap interval for the concordance statistic.
- [Section 2, Results] The claim that "Pillar-0 never saw CT-RATE yet remains in the top three on the two held-out cohorts" would be stronger with a statement about possible data contamination in the pretraining corpora of all models; consider discussing this.
Circularity Check
No significant circularity: the paper is an empirical benchmark whose central claims are measured independently of their inputs, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain.
full rationale
No circular step can be identified from the text. The paper performs an empirical evaluation of ten frozen CT encoders on three cohorts; it does not fit a parameter to a subset of data and then predict a closely related quantity. The central claim that detectability scales with contrast and spatial extent is inferred from AUROC values measured across ten encoders and multiple readouts, including an unseen internal cohort, and the empirical difficulty ordering is quantified rather than assumed. The within-organ comparisons in Fig. 4 have validity concerns—contrast and extent are assigned qualitatively, and the pairs differ in prevalence and label-extraction noise—but this is a confounding/underdetermination issue, not circularity: the AUROC values are not constructed from the physical labels. The paper also explicitly acknowledges the label-noise limitation and attempts to rebut it with Kendall W = 0.89 across independently constructed label pipelines. Self-citations appear (e.g., the RATE framework for internal-cohort labels, and Lungevaty as a spatial-localization comparison), but they are not load-bearing for the central claim, and the benchmark's result that CT-CLIP underperforms cuts against a self-serving bias. No uniqueness theorem, ansatz, or first-principles derivation is imported from the authors' prior work. The causal attribution to globally pooled embeddings is an interpretation that goes beyond the data, but that is a scientific-scope concern rather than a circular reduction.
Assumptions & free parameters
free parameters (4)
- k in kNN =
5
- C in linear probe =
1
- Minimum positive count =
20
- Within-organ comparison pairs =
7 pairs
assumptions (4)
- domain assumption The CT-RATE validation split is patient-disjoint from CT-RATE training data used by five models
- domain assumption NLP-extracted labels are accurate enough that label noise does not drive the focal-lesion failure
- domain assumption The within-organ pairs differ only along the intended axis (contrast or extent)
- ad hoc to paper Global average pooling is the mechanism behind the focal-lesion failure
Cite this review
Pith. "Pith review of Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models." pith.science (2026). https://pith.science/paper/4P5LCZZ4
@misc{pith2026260805960,
author = {Pith},
title = {Pith review of: Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/4P5LCZZ4}},
note = {Machine review of arXiv:2608.05960}
}
abstract
Routine CT interpretation is inherently comprehensive, capturing incidental findings across the entire scan volume. 3D CT foundation models could assist this process by providing generalizable representations of anatomy and pathology. To evaluate their diagnostic breadth, we benchmark ten frozen CT encoders across three cohorts of thoracic CT scans, including an unseen internal clinical dataset, using $k$-nearest neighbors, zero-shot prompting, and linear probing. We find no universal state-of-the-art, with rankings fluctuating significantly depending on the evaluation context. While models combining fine-grained image tokenization with vision-language alignment generally perform best, a lightweight supervised encoder remains highly competitive, demonstrating that explicit labels can effectively substitute for scale. Crucially, rather than model architecture, we observe that the primary determinant of performance is a physical bottleneck: a finding's detectability scales with its contrast against surrounding tissue and its spatial extent. Through controlled within-organ comparisons, we empirically demonstrate that widespread or high-contrast abnormalities, such as devices and effusions, are reliably recovered. Conversely, small, low-contrast focal lesions remain a persistent challenge across all evaluated encoders. We attribute this to the inherent limitations of globally pooled embeddings, suggesting that accurately representing small, low-contrast structures will require region- or lesion-level pretraining.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Aerts, H.J.W.L., Pai, S., Hadzic, I., Fedorov, A., Mak, R.H.: Foundation model embeddings for quantitative tumor imaging biomarkers. Research Square (2025). https://doi.org/10.21203/rs.3.rs-6630446/v1, preprint rs.3.rs-6630446
-
[2]
arXiv preprint arXiv:2511.17803 (2025)
Agrawal, K.K., Liu, L., Lian, L., Nercessian, M., Harguindeguy, N., et al.: Pillar-0: a new frontier for radiology foundation models. arXiv preprint arXiv:2511.17803 (2025)
arXiv 2025
-
[3]
Japanese Journal of Radiology43(10), 1606– 1617 (2025)
Aydin, S., Ece, B., Cakmak, V., Kocak, B., Onur, M.R.: Emergency radiology: roadmap for radiology departments. Japanese Journal of Radiology43(10), 1606– 1617 (2025)
work page 2025
-
[4]
arXiv preprint arXiv:2406.06512 (2024)
Blankemeier, L., Cohen, J.P., Kumar, A., Van Veen, D., Gardezi, S.J.S., et al.: Merlin: a vision language foundation model for 3d computed tomography. arXiv preprint arXiv:2406.06512 (2024)
arXiv 2024
-
[5]
In: 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI)
Brandt, J., Chevli, M., Braren, R., Kaissis, G., Müller, P., Rueckert, D.: Lungevaty: A scalable, open-source transformer-based deep learning model for lung cancer risk prediction in ldct screening. In: 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI). pp. 1–5. IEEE (2026) 10 M. Chevli, J. Brandt et al
work page 2026
-
[6]
arXiv preprint arXiv:2511.17209 (2025)
Claessens, C., Viviers, C., D’Amicantonio, G., Bondarev, E., van der Sommen, F.: Scaling self-supervised and cross-modal pretraining for volumetric ct transformers. arXiv preprint arXiv:2511.17209 (2025)
arXiv 2025
-
[7]
Di Piazza, T., Lazarus, C., Nempont, O., Boussel, L.: Structured spectral graph representation learning for multi-label abnormality analysis from 3d ct scans. arXiv preprint arXiv:2510.10779 (2025)
work page Pith review arXiv 2025
-
[8]
Medical Image Analysis67, 101857 (2021)
Draelos, R.L., Dov, D., Mazurowski, M.A., Lo, J.Y., Henao, R., Rubin, G.D., Carin, L.: Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes. Medical Image Analysis67, 101857 (2021). https://doi.org/10.1016/j.media.2020.101857
arXiv 2021
Show all 20 references
-
[9]
Annals of emergency medicine80(3), 243–256 (2022)
Evans, C.S., Arthur, R., Kane, M., Omofoye, F., Chung, A.E., Moreton, E., Moore, C.: Incidental radiology findings on computed tomography studies in emergency department patients: a systematic review and meta-analysis. Annals of emergency medicine80(3), 243–256 (2022)
2022
-
[10]
arXiv preprint arXiv:2403.17834 (2024)
Hamamci, I.E., Er, S., Almas, F., Simsek, A.G., Esirgun, S.N., et al.: Developing generalist foundation models from a multimodal dataset for 3d computed tomog- raphy. arXiv preprint arXiv:2403.17834 (2024)
2024
-
[11]
Elsevier Health Sciences (2019)
Herring, W.: Learning radiology: recognizing the basics. Elsevier Health Sciences (2019)
2019
-
[12]
Research Square (2026)
Hoelzle, C.S., Brandt, J., Mueller, J.C., Klug, M., Westphal, J., Rueckert, D., Chevli, M., Fintelmann, F.J.: Longitudinal language-model reasoning enables au- tomated labeling of lung cancer recurrence from unstructured clinical records. Research Square (2026). https://doi.or...
2026 doi
-
[13]
arXiv preprint arXiv:2605.21906 (2026)
Li, Y., Gao, Y., Dong, H., Lai, Y., Wang, S., Safari, M., Baciak, J.E., Yang, X.: Universal ct representations from anatomy to disease phenotype through agglom- erative pretraining. arXiv preprint arXiv:2605.21906 (2026)
2026 arXiv
-
[14]
Radiology312(3), e240541 (2024)
Marcinkiewicz, A.M., Buchwald, M., Shanbhag, A., Bednarski, B.P., Killekar, A., Miller, R.J.H., Builoff, V., Lemley, M., Berman, D.S., Dey, D., Slomka, P.J.: AI for multistructure incidental findings and mortality prediction at chest CT in lung can- cer screening. Radiology312...
2024 doi
-
[15]
Journal of Clinical Oncology41(12), 2191–2200 (2023)
Mikhael, P.G., Wohlwend, J., Yala, A., Karstens, L., Xiang, J., Takigami, A.K., Bourgouin, P.P., Chan, P., Mrah, S., Amayri, W., Juan, Y.H., Yang, C.T., Wan, Y.L., Lin, G., Sequist, L.V., Fintelmann, F.J., Barzilay, R.: Sybil: A validated deep learning model to predict future ...
2023
-
[16]
arXiv preprint arXiv:2604.04133 (2026)
Moreno-Aguado, R., Magallón, A., Moreno, V., Fang, Y., Yang, G.: Learning ro- bust visual features in computed tomography enables efficient transfer learning for clinical tasks. arXiv preprint arXiv:2604.04133 (2026)
2026 arXiv
-
[17]
arXiv preprint arXiv:2501.09001 (2025)
Pai, S., et al.: Vision foundation models for computed tomography. arXiv preprint arXiv:2501.09001 (2025)
2025 arXiv
-
[18]
arXiv preprint arXiv:2604.01987 (2026)
Saporta,A.,Callard,B.,Dancette,C.,Khlaut,J.,Corbière,C.,Butsanets,L.,Prat, A., Manceron, P.: Curia-2: scaling self-supervised learning for radiology foundation models. arXiv preprint arXiv:2604.01987 (2026)
2026
-
[19]
arXiv preprint arXiv:2510.15042 (2025)
Wald, T., Hamamci, I.E., Gao, Y., Bond-Taylor, S., et al.: Comprehensive language–image pre-training for 3d medical image understanding. arXiv preprint arXiv:2510.15042 (2025)
2025
-
[20]
IEEE transactions on pattern analysis and machine intelligence45(9), 10795– 10816 (2023)
Zhang, Y., Kang, B., Hooi, B., Yan, S., Feng, J.: Deep long-tailed learning: A sur- vey. IEEE transactions on pattern analysis and machine intelligence45(9), 10795– 10816 (2023)
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.