{"id":"97ecfec2-c881-4cd7-84a6-ca1d24827014","arxiv_id":"2608.05960","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Across ten frozen 3D CT encoders, a finding's detectability is governed by its contrast and spatial extent, not the model choice, and small low-contrast lesions remain undetectable even with linear probing.","lead":"This paper benchmarks ten 3D CT foundation models on three chest CT datasets and finds that no single model wins in every setting. Across all models, large or high-contrast abnormalities are detected reliably while small, low-contrast lesions are consistently missed, which the authors trace to globally pooled embeddings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal claim that globally pooled embeddings are the bottleneck is not established: the seven within-organ pairs also differ in label noise and prevalence, and contrast/extent are assigned qualitatively, so the difficulty ordering may reflect measurement artifacts rather than embedding geometry.","rationale":"The paper is a careful benchmark, and the consistent difficulty ordering across encoders and cohorts is a real empirical contribution. The reader's verdict of CONDITIONAL is appropriate: the central causal claim goes beyond the data. My stress-test focuses on the specific mechanism by which the claimed physical bottleneck could be an artifact: label noise and prevalence differences in the seven within-organ pairs. This is not a question of model dishonesty or internal inconsistency; it is a question of whether the contrast/extent axis is identified or merely correlated with other properties of the findings and their labels. The proposed manual re-review test is feasible on a public cohort and would directly settle whether the AUROC gaps survive cleaned labels. If they do, the physical-bottleneck claim gains real support; if they do not, the paper's recommendation for region-level pretraining would need to be reframed as a hypothesis rather than a conclusion. Since the reader already flagged the same family of confounds, my read does not change the verdict.","tokens_in":11744,"tokens_out":3918,"duration_ms":41149,"concrete_test":"Take one public cohort (e.g., CT-RATE) and have a radiologist manually re-review a stratified sample of roughly 50-100 scans per finding type in the seven Fig. 4 pairs, producing clean gold labels; recompute the Fig. 4 macro-AUROC gaps on prevalence-matched subsets. If the contrast and extent gaps persist under clean labels and matched prevalence, the physical-bottleneck interpretation survives; if they shrink to non-significance, label noise or prevalence, not global pooling, explains the ordering.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that detectability is governed by the physical properties contrast and extent, and specifically that globally pooled embeddings cannot represent small low-contrast lesions. The evidence for this is Fig. 4's seven within-organ comparisons, but those comparisons do not isolate the physical axes. Contrast and extent are assigned qualitatively, and the pairs differ in prevalence, label-extraction pipeline, and radiological definition. The most damaging confound is label noise: all three cohorts derive labels from radiology reports via NLP or rule-based extraction, and subtle findings such as subsolid nodules and ground-glass opacities are systematically harder to extract and less consistently mentioned by radiologists. Since AUROC is measured against these noisy labels, a low AUROC for subtle findings can reflect a label ceiling rather than a property of the embeddings. The paper's robustness argument (Kendall W = 0.89 across independently constructed pipelines) does not rule this out: all pipelines share the same underlying reports, whose language is less reliable for exactly the findings the paper calls invisible. Moreover, the paper never compares global-pooling encoders against region- or lesion-level models, so the attribution to pooling rather than label noise or training objective is not directly tested. The empirical difficulty ordering is credible, but the causal mechanism is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks ten frozen 3D CT encoders on three thoracic CT datasets (CT-RATE, RAD-ChestCT, and an unseen internal hospital cohort) using three readouts (kNN, zero-shot prompting, and linear probing), reporting AUROC and prevalence-normalized skill. It finds no universal state-of-the-art, with fine-grained tokenization plus vision–language alignment generally leading but a lightweight supervised encoder remaining competitive. The central empirical observation is a consistent difficulty ordering across encoders and cohorts (Kendall W = 0.89): devices and fluid are easy, focal low-contrast lesions are hard. The authors attribute this to a physical bottleneck (contrast and spatial extent) and conclude that globally pooled embeddings cannot adequately represent small, low-contrast structures, recommending region- or lesion-level pretraining.","tokens_in":11982,"tokens_out":4738,"duration_ms":44643,"significance":"The benchmark is valuable in several respects: it evaluates a truly unseen clinical cohort, uses patient-grouped cross-validation and bootstrap confidence intervals, covers three distinct readouts, and provides code for the public cohorts. The consistent difficulty hierarchy across ten encoders and three cohorts is a robust empirical contribution that can inform model selection and evaluation design, independent of whether the causal explanation is accepted. The paper also gives credit to the possibility that explicit labels can substitute for scale, a finding that is interesting for practitioners. The weakness is the gap between the observed correlation and the causal claim about global pooling, which is not directly tested. If the causal claim is confirmed, the paper would have significant implications for pretraining objectives.","major_comments":[{"comment":"The within-organ comparisons are presented as isolating contrast and spatial extent, but the pairs differ in multiple other dimensions beyond the claimed physical axis. Contrast and extent are assigned qualitatively, and the pairs also differ in prevalence, label-extraction pipeline, and radiological definition (e.g., pleural effusion vs. thickening; solid vs. subsolid nodule). The bootstrap-significant AUROC differences and the 68/70 directionality agreement establish a highly consistent ordering, but not that the intended physical properties are the cause. To support the causal claim, the authors should quantify contrast (e.g., HU difference to surrounding tissue) and extent (e.g., lesion volume or segmentation) for each pair, and provide a per-pair table reporting prevalence and label source, so that these potential confounds can be assessed or adjusted.","section":"Section 4, Finding Types; Fig. 4"},{"comment":"The central attribution to \"globally pooled embeddings\" is not directly tested. No experiment compares these frozen encoders against region-level or lesion-level features under the same protocol, and no ablation removes pooling (e.g., by using patch embeddings from the same encoders instead of the global vector). The cited spatial-localization systems (Sybil, LungEvaty) are not evaluated here. As written, the conclusion is a hypothesis consistent with the data rather than a demonstrated mechanism. The authors should either add a comparison or ablation that directly addresses pooling, or rephrase the conclusion as a testable prediction, noting the absence of direct evidence.","section":"Section 3, Discussion; Abstract"},{"comment":"The label-noise concern is acknowledged but dismissed based on the consistency of the difficulty ordering across independently constructed pipelines (Kendall W = 0.89). This does not rule out the confound: all three pipelines derive labels from radiology reports, whose language is systematically less reliable for subtle findings (subsolid nodules, ground-glass opacities), so the shared report-language bias could produce the same ordering even if the embeddings were insensitive to focal lesions. Provide a control using adjudicated labels or label-confidence scores (e.g., from the NLP extractors) to show that the ordering persists when label noise is reduced, or explicitly account for label uncertainty in the analysis.","section":"Section 4, Datasets; Discussion, Limitations"}],"minor_comments":[{"comment":"The term \"globally pooled embeddings\" is used for all ten models, but a few models use graph/CNN pooling or other aggregation schemes. Specify the actual pooling mechanism for each encoder, or restrict the claim to models with global average pooling.","section":"Section 4, Models"},{"comment":"Figure 5 is very dense and nearly illegible at column width; consider splitting it per cohort or enlarging the font.","section":"Fig. 5"},{"comment":"The dataset name is inconsistently written as \"RadChestCT\", \"RAD-Chest CT\", and \"RAD-ChestCT\"; unify the naming.","section":"Throughout"},{"comment":"Kendall's W is reported without a confidence interval or p-value; report a bootstrap interval for the concordance statistic.","section":"Section 2, Results"},{"comment":"The claim that \"Pillar-0 never saw CT-RATE yet remains in the top three on the two held-out cohorts\" would be stronger with a statement about possible data contamination in the pretraining corpora of all models; consider discussing this.","section":"Section 2, Results"}],"recommendation":"major_revision","confidential_remarks":"The authors appear to be affiliated with several evaluated models and datasets (CT-CLIP, COLIPRI, CT-RATE), yet the manuscript does not declare a conflict of interest or describe the involvement of the respective model authors in the benchmark design. This should be declared in the revised version. The fit to the journal's scope is appropriate, and the empirical benchmark is useful; the main revision needed is a more careful separation of the robust empirical ordering from the causal interpretation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this benchmark is carefully done and the main empirical finding holds up; the headline causal claim does not. The paper's real result is that the difficulty of detecting a finding in frozen CT embeddings is remarkably consistent across ten very different encoders — small low-contrast focal lesions fail everywhere, and Kendall W=0.89 across encoders, with the ordering replicating on three cohorts including an unseen internal one. That is a useful, credible empirical regularity, and the paper earns credit for patient-grouped CV, bootstrap CIs, three independent readouts, and shipping code for the public cohorts. The data-efficiency curves (Pillar-0 saturating, the small supervised CT-SSG staying competitive) are also genuinely informative for model selection.\n\nThe soft spot is exactly where the reader put it: the jump from \"small low-contrast lesions score low\" to \"globally pooled embeddings are the bottleneck.\" That mechanism is not tested. There is no region- or lesion-level encoder in the comparison, so \"pooling discards the signal\" is never distinguished from \"pretraining never captured it\" or \"the NLP label ceiling for subtle findings is what we're measuring.\" The seven within-organ pairs are a decent attempt to control anatomy, but contrast and extent are assigned qualitatively, and the pairs also differ in prevalence and in how reliably the report extractors catch the subtle member. The Kendall-W argument against label noise is weaker than the paper claims: all three pipelines read the same kind of reports, and subtle findings are exactly the ones radiologists mention inconsistently. I'd hold the middle ground: the empirical regularity is real, the mechanism is underdetermined, and the stress-test's \"measurement artifacts rather than embedding geometry\" is too dismissive — a pure artifact story would struggle to produce such a clean contrast/extent ordering across independent cohorts.\n\nTwo smaller points. The authors evaluate several models from their own group (COLIPRI, CT-CLIP, CT-RATE) without a disclosure statement; minor, but it should be added. And the abstract's \"invisible\" framing overreaches slightly — Fig. 3b itself shows probe > zero-shot > kNN, so some signal is present but not language-reachable.\n\nWho's it for: anyone choosing a chest CT encoder or designing next-round pretraining objectives. It deserves a serious referee. The right request is to soften the causal claim or test it directly with a non-pooled readout on the same cohorts. I'd bring it to reading group.","headline":"A careful, reproducible benchmark whose empirical difficulty-ordering result is credible and useful; the global-pooling causal story is a plausible inference, not a demonstrated mechanism.","tokens_in":12533,"tokens_out":4470,"would_cite":true,"duration_ms":43234,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen 3D CT encoder's ability to detect a finding is set by the finding's contrast and spatial extent, not by the model's architecture.","keywords":["Foundation models","Chest CT","Incidental findings","Multi-abnormality classification","Frozen features","Transfer evaluation","Contrast and extent","Global pooling"],"falsifier":"Measure the actual Hounsfield-unit contrast and volumetric extent of every finding in the seven comparison pairs and check whether AUROC differences track the measured values; alternatively, insert synthetic lesions of fixed contrast and varying size into scans and recover them with a frozen encoder. If small bright lesions are recovered as easily as large bright ones, the extent axis of the claim fails; if large low-contrast lesions are missed, the contrast axis fails.","tokens_in":11517,"feed_emoji":"🩻","tokens_out":6628,"duration_ms":61720,"temperature":0.7,"pith_summary":"This paper asks whether frozen 3D CT foundation models, used without any fine-tuning, can detect the broad range of incidental findings a radiologist would spot, and where their limits lie. Benchmarking ten encoders on three thoracic CT cohorts with k-nearest-neighbour, zero-shot, and linear-probe readouts, it finds no single best model; rankings shift by readout. The consistent result is physical: detectability scales with a finding's contrast against surrounding tissue and its spatial extent. Devices and effusions are reliably recovered, while small low-contrast focal lesions such as subsolid nodules stay near chance for every encoder, a limit the paper attributes to globally pooled embeddings.","feed_headline":"Small low-contrast CT lesions stay invisible to every model","feed_subtitle":"Across 3 cohorts and 10 encoders, detection tracks a finding's size and contrast, so small lesions need new pretraining.","key_machinery":"The load-bearing object is the global pooled embedding: a single vector formed from the entire scan volume, typically by aggregating patch or token features, which is what all three readouts consume. The analysis that isolates the paper's physical claim is the set of seven within-organ comparison pairs, two findings in the same organ matched to differ mainly in contrast or mainly in extent, measured by macro AUROC averaged over ten encoders. The readout decomposition (cosine k-NN, zero-shot text prompting, linear probing) shows that the difficulty hierarchy is not an artifact of a weak readout, because the linear probe extracts more signal yet still fails on focal lesions.","core_discovery":"On the paper's own terms, the central claim is that a finding's detectability in a frozen 3D CT encoder is governed by two physical properties, contrast against surrounding tissue and spatial extent, rather than by model architecture or scale. This is shown with controlled within-organ comparisons: holding extent fixed while raising contrast (effusion versus thickening, solid versus subsolid nodule) improves macro AUROC by about +0.18, and holding contrast fixed while increasing extent (bulk versus faint calcification, diffuse versus focal ground-glass) improves it by about +0.16. All seven pairs show the same direction of effect with bootstrap significance, and 68 of 70 encoder-pair comparisons agree. The same easy-to-hard ordering appears across an unseen internal cohort and across k-NN, zero-shot, and linear-probe readouts, so the paper concludes that globally pooled embeddings inherently attenuate the small spatial signals of focal lesions and that those lesions will remain unrecoverable unless future pretraining uses region- or lesion-level objectives.","pith_inferences":["The same physical ceiling should appear in any volumetric medical-imaging encoder that pools globally, including MRI and PET, so a direct test in those modalities would show whether the mechanism generalizes.","One testable extension is to pretrain with a contrastive loss that operates on local patches or detected regions and compare against a global-pooling baseline on small lesions; the paper's account predicts the local objective should break the ceiling.","The consistency across three independently labeled cohorts suggests label noise is not the driver, but a causal version of the claim would need measured Hounsfield-unit contrast and lesion volume rather than qualitative assignment for each comparison pair.","The near-universal directionality across encoders implies that gain from better architecture concentrates on easy findings, so aggregate leaderboards may mislead model selection for radiology workflows."],"forward_implications":["If the claim holds, clinicians should not rely on frozen global CT embeddings for small low-contrast incidental findings; those findings need spatially grounded detection.","Benchmarking should report per-finding-type results rather than a single macro AUROC, because global scores hide the physical ceiling.","Pretraining objectives that operate on regions or lesions, rather than whole volumes, are the direct route to closing the focal-lesion gap.","Fine-grained tokenization and vision-language alignment improve retrieval and zero-shot ability but do not remove the physical bottleneck; explicit labels can substitute for scale on easy findings but not on small low-contrast ones.","For rare findings, the small zero-shot-over-kNN AUROC advantage vanishes under prevalence-normalized skill, so readout choice matters mainly for common abnormalities."],"supporting_citations":[{"why":"Supplies the CT-RATE public cohort and the vision-language CT-CLIP model; five evaluated encoders were pretrained on this data, making it central to the benchmark's leakage-controlled design.","marker":"[10]"},{"why":"Supplies the RAD-ChestCT public cohort with 92 derived labels, providing the second large evaluation set used for the physical-bottleneck comparisons.","marker":"[8]"},{"why":"Provides the RATE label-extraction framework used for the unseen internal hospital cohort and contributes the Pillar-0 encoder, which is central to the label-extraction and architectural comparisons.","marker":"[2]"},{"why":"Introduces CT-SSG, the supervised encoder whose competitiveness supports the paper's claim that explicit labels can substitute for scale.","marker":"[7]"},{"why":"Introduces FlexiCT, one of the leading vision-language encoders with fine-grained tokenization, anchoring the comparison of architectural traits.","marker":"[13]"},{"why":"Introduces COLIPRI, the other leading vision-language encoder, which anchors the claim that fine-grained tokenizers combined with report alignment drive top retrieval performance.","marker":"[19]"}],"fun_headline_variants":["Size and contrast, not architecture, dictate CT lesion detectability","Small low-contrast lesions stay elusive for every 3D CT encoder","Big or bright? That's what CT foundation models actually see","Contrast and extent, not model choice, drive CT detection success","Frozen CT encoders: brightness and bulk matter more than scale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the seven hand-picked within-organ comparison pairs isolate contrast or extent from everything else; those pairs also differ in prevalence, label-extraction pipeline, and radiological definition, so if this isolation fails, the physical-bottleneck claim reduces to saying conspicuous findings are conspicuous.","fun_headline_variants_meta":{"raw":{"variants":["Size and contrast, not architecture, dictate CT lesion detectability","Small low-contrast lesions stay elusive for every 3D CT encoder","Big or bright? That's what CT foundation models actually see","Contrast and extent, not model choice, drive CT detection success","Frozen CT encoders: brightness and bulk matter more than scale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000615,"raw_usage":{"total_tokens":2877,"prompt_tokens":987,"completion_tokens":1890,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1799}},"tokens_in":603,"tokens_out":1890,"duration_ms":12090,"temperature":1.0,"reasoning_tokens":1799,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:17:04.343470+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the actual Hounsfield-unit contrast and volumetric extent of every finding in the seven comparison pairs and check whether AUROC differences track the measured values; alternatively, insert synthetic lesions of fixed contrast and varying size into scans and recover them with a frozen encoder. If small bright lesions are recovered as easily as large bright ones, the extent axis of the claim fails; if large low-contrast lesions are missed, the contrast axis fails.","supporting_citations":[{"cited_title":"Structured Spectral Graph Representation Learning for Multi-label Abnormality Analysis from 3D CT Scans","cited_arxiv_id":"2510.10779","evidence_quote":"Introduces CT-SSG, the supervised encoder whose competitiveness supports the paper's claim that explicit labels can substitute for scale."},{"cited_title":"Universal CT Representations from Anatomy to Disease Phenotype through Agglomerative Pretraining","cited_arxiv_id":"2605.21906","evidence_quote":"Introduces FlexiCT, one of the leading vision-language encoders with fine-grained tokenization, anchoring the comparison of architectural traits."}],"review_version":1}