REVIEW 3 major objections 6 minor 32 references
CXR-CML: Improved zero-shot classification of long-tailed multi-label diseases in Chest X-Rays
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read CXR-CML claims that re-clustering CLIP's chest X-ray latent space with a Student-t mixture and a triplet loss raises 40-class zero-shot AUC from 0.644 to 0.715, lifting rare-class AUC from 0.631 to 0.720.
desk verdict The zero-shot claim is not supported by the experimental protocol; what remains is an incremental supervised fine-tuning result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a three-stage latent-space refiner on top of CLIP: (1) a Gaussian Mixture Model with $N=40$ components soft-clusters the visual embeddings; (2) a Student t-distribution mixture with $\nu=4$ reweights the assignments so tail samples keep non-negligible membership; (3) a triplet loss $L_m = \sum \max(0, d(a,p)-d(a,n)+\alpha)$ trained jointly with CLIP's contrastive loss $L_c$ pulls same-cluster embeddings together and pushes different clusters apart, using the cluster assignments as pseudo-labels. Text 'meta-labels' generated from ground-truth disease names supply the supervision that aligns these clusters with semantics. The degrees of freedom $\nu$ and batch size are the key tuning knobs.
What would settle it
Retrain the pipeline with the GMM, Student-t, and triplet-loss components disabled while keeping the meta-label text supervision and all other hyperparameters the same; if the 40-class total AUC stays near the CheXzero-plus-meta-labels level of 0.691 instead of reaching 0.715, the proposed clustering machinery is not what produces the reported gain.
Extended reading notes
Core claim
The central claim is that the failure of vision-language models on long-tailed chest X-ray diseases is a representation-geometry problem, not just a data-scarcity problem. CLIP's shared image-text space places rare classes in overlapping regions of a single Gaussian-shaped manifold; CXR-CML instead fits a Gaussian mixture model to the visual features, refines the mixture with a Student t-distribution whose heavy tails assign nonzero probability to rare instances, and then applies a triplet loss that uses the refined cluster assignments as pseudo-labels. The paper reports that this pipeline raises total macro AUC from 0.644 to 0.715 over 40 classes, and that rare classes (those shown in fewer than 1,000 training images) actually exceed base-class AUC (0.720 vs 0.711), reversing the usual long-tail deficit.
Load-bearing premise
The central claim depends on the assumption that the AUC gains come from the GMM, Student-t, and triplet-loss components, not from the extra supervised signal provided by meta-labels generated from the target dataset's ground-truth labels.
Editorial extensions
If this is right
- If the reported AUCs hold, the 12 rare classes would no longer be the weak point of CXR classification; the method reports rare AUC above base AUC.
- The 40-class CXR-LT benchmark would become the standard evaluation for zero-shot CXR models, since 14-class evaluations hide long-tail failures.
- The same GMM-plus-t-distribution-plus-metric-loss recipe could be lifted onto any CLIP-style encoder for other long-tailed medical image domains.
- Because the method fine-tunes CLIP on the target dataset with generated text, the practical claim is for semi-supervised adaptation, not annotation-free deployment.
Reading between the lines
- The paper's framing calls the evaluation zero-shot, but the pipeline is trained on MIMIC-CXR-JPG ground-truth labels to generate meta-labels; treating the reported 0.715 as a true zero-shot number would overstate what a no-labels deployment would achieve.
- The ablation attributes most of the total gain to meta-label supervision alone (0.644 to 0.691), with the GMM/t/triplet components adding a further ~0.024; a decomposed ablation that holds meta-labels fixed is needed to isolate the claimed mechanism.
- The Student-t degrees of freedom $\nu$ and the GMM component count $N$ are both chosen by validation; one could link $N$ to the known label count or infer it with a Dirichlet process to remove that tuning.
- Applying the same text-template style to other imbalanced radiology datasets (e.g., mammography or CT reports) would be a direct test of whether the improvement transfers.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CXR-CML, a method built on CLIP for long-tailed multi-label chest X-ray classification. The authors apply Gaussian Mixture Model (GMM) clustering to CLIP visual embeddings, claim to refine the clusters with a Student t-distribution, and add a triplet loss using GMM-derived pseudo-labels. The total loss is L = Lc + Lm, with text captions generated from MIMIC-CXR-JPG ground-truth annotations. The method is evaluated on 40 classes from the MICCAI CXR-LT challenge, reporting a total AUC of 0.715 versus a CheXzero baseline of 0.644, with the abstract claiming a 7% (and later 2%) improvement in zero-shot AUC. An ablation shows that adding ground-truth-derived meta labels to CheXzero raises total AUC from 0.644 to 0.691.
Significance. If the reported gains were genuinely zero-shot and attributable to the proposed clustering mechanism, the method would be a useful contribution to long-tailed medical image classification. The paper does include some strengths: 5-fold cross-validation, multiple runs, and an ablation study that attempts to separate the contribution of the meta labels. However, the central claim is undermined by the experimental protocol, which uses ground-truth labels from the target dataset for fine-tuning, and by the fact that the Student t-distribution refinement is never algorithmically specified. The additional supervised signal from the meta labels explains most of the observed improvement, so the proposed mechanism is not demonstrated. The paper also contains internal inconsistencies in reported numbers and experimental details.
major comments (3)
- [§2 Text Generation; §3 Experimental Setup] The reported evaluation is not zero-shot. Section 2 states that 'Textual descriptions are constructed for all classes that are marked as present in the groundtruth annotations,' and Section 3 states that text is generated from the MICCAI annotation data and that the model is trained on a training split derived from the same dataset. The full loss L = Lc + Lm in Eq. (4) is optimized on MIMIC-CXR-JPG ground-truth-derived captions, and results are reported on the validation split. A zero-shot baseline such as CheXzero uses no training on the target dataset, so the CXR-CML row in Table 2 cannot be compared with it as a zero-shot method. The abstract's claim of '7% points in zero-shot AUC' is therefore unsupported.
- [§2 Modeling the distribution: Student t-distribution, Eq. (2)] The proposed Student t-distribution refinement is never algorithmically specified. The text gives only the mixture density in Eq. (1) and the t-density in Eq. (2), followed by the statement that the model 'refine[s] them using a Student t-distribution.' There is no update rule, objective function, or pseudocode explaining how the t-distribution parameters are estimated, how the GMM clusters are refined into t-mixture clusters, or how the pseudo-labels used in Eq. (3) are derived. Without this, the core mechanism attributed for the reported gains cannot be reproduced or verified.
- [Table 3, Ablation Study] The ablation attributes most of the gain to the ground-truth-derived meta labels rather than to the proposed clustering components. CheXzero + Meta labels reaches 0.691 total AUC, while the full CXR-CML reaches 0.715, so only 0.024 AUC is left for the GMM, Student-t, and triplet-loss contributions. No ablation removes the meta labels or adds the GMM/t-distribution components one at a time, so the paper cannot support the claim that the proposed mechanism, rather than the additional supervised signal, is responsible for the improvement.
minor comments (6)
- [Abstract] The abstract contains inconsistent improvement numbers: it says '7% points' in one sentence and '2%' in another; these should be reconciled with the results in Table 2.
- [§3 Dataset] The first sentence says the dataset is 'labeled with 39 disease classes,' but the rest of the paper consistently refers to 40 classes; please correct the count.
- [Abstract, code availability] The code link is given as the placeholder text 'CXR-CML' rather than an actual URL, so the reproducibility claim cannot be checked.
- [§4 Results, Table 3] Table 3 is referenced in the text as 'Table ??' and contains a duplicated row for 'CXR-CML (bs = 32, ν = 4)'; please fix the reference and remove the duplicate.
- [§3 Implementation Details and Table 2 caption] The number of runs is inconsistent: Table 2 says results are 'averaged over 10 runs,' while the Implementation Details say 'the average of 5 runs'; please clarify the actual protocol.
- [Throughout] There are several typos and formatting issues, including 'Thererfore,' 'T ext Generation,' 'L⌋', 'comparitive,' and inconsistent use of 'Student t-distribution' versus 'Student’s t-distribution'; these should be corrected in revision.
Circularity Check
The reported gain is not zero-shot: §2 fine-tunes CLIP on target ground-truth-derived captions, so the claimed 7% zero-shot AUC improvement is a supervised result relabeled as zero-shot.
-
fitted input called prediction
[§2 Text Generation; §3 Experimental Setup; Table 2]
"Textual descriptions are constructed for all classes that are marked as present in the groundtruth annotations. ... The network is trained with the complete loss which is given as, L = Lc + Lm. ... we split the dataset into training and test sets using an 80:20 ratio. ... Table 2 indicates the 5-fold average AUC scores for all, base and rare classes on the validation set."
Zero-shot classification is defined by the absence of training on the target dataset's labels. Here the text captions used to train CLIP are generated from the ground-truth labels of MIMIC-CXR-JPG, and the full model is optimized with L = Lc + Lm on that dataset's training split. The CXR-CML row in Table 2 is therefore a supervised fine-tuning result, not a zero-shot prediction. Comparing it to zero-shot baselines means the claimed '7% points in zero-shot AUC' is a supervised result renamed as a zero-shot prediction rather than a prediction derived from the proposed GMM/t-distribution/triplet mechanism.
full rationale
The only step that reduces to its own input is the labeling of the fine-tuned result as zero-shot. The GMM/t-distribution/triplet machinery is not circular in itself: the pseudo-labels are derived from the model's own embeddings and used for self-training, which is a recognized technique, and no equation in the paper makes the held-out AUC equal to the training loss. The self-citation '[anon]' is anonymized and not load-bearing, and the math in Eqs. (1)-(4) is standard. The ablation (Table 3) shows that CheXzero + Meta labels accounts for 0.691 of the 0.715 total, so the attribution of the gain to GMM/t-distribution/triplet is not demonstrated; that is a correctness/control problem, not a circularity. The central circular/definitional defect is the zero-shot claim: since training captions are built from the target dataset's ground-truth labels and the full model is optimized on the target training split, the reported 'zero-shot AUC' is by definition a supervised result. Score 6 because the central claim's qualifier 'zero-shot' is invalidated by the method's own inputs, even though the underlying supervised numbers are empirical.
Assumptions & free parameters
free parameters (5)
- number of GMM components N =
40
- Student-t degrees of freedom ν =
4
- triplet margin α =
not reported
- batch size =
32
- loss weights for Lc and Lm =
equal (1:1)
assumptions (4)
- domain assumption CLIP embeddings are a meaningful manifold for CXR disease classification
- ad hoc to paper GMM with N=40 components approximates the true latent distribution
- ad hoc to paper Student t-distribution refinement improves clustering and is correctly implemented
- domain assumption Generated text from ground-truth labels is a valid weak supervision signal
Cite this review
Pith. "Pith review of CXR-CML: Improved zero-shot classification of long-tailed multi-label diseases in Chest X-Rays." pith.science (2026). https://pith.science/paper/ITL7ZWCX
@misc{pith2026250719398,
author = {Pith},
title = {Pith review of: CXR-CML: Improved zero-shot classification of long-tailed multi-label diseases in Chest X-Rays},
year = {2026},
howpublished = {\url{https://pith.science/paper/ITL7ZWCX}},
note = {Machine review of arXiv:2507.19398}
}
read the original abstract
Chest radiography (CXR) plays a crucial role in the diagnosis of various diseases. However, the inherent class imbalance in the distribution of clinical findings presents a significant challenge for current self-supervised deep learning models. These models often fail to accurately classify long-tailed classes. Current Vision-Language models such as Contrastive Language Image Pre-training (CLIP) models effectively model the manifold distribution of the latent space, enabling high zero-shot classification accuracies. Although CLIP performs well on most of the primary classes in the dataset, our work reveals that its effectiveness decreases significantly for classes with a long-tailed distribution. Our approach employs a class-weighting mechanism that directly aligns with the distribution of classes within the latent space. This method ensures a substantial improvement in overall classification performance, with particular emphasis on enhancing the recognition and accuracy of rarely observed classes. We accomplish this by applying Gaussian Mixture Model (GMM) clustering to the latent space. The subsequent clusters are further refined by Student t-distribution, followed by a metric loss that utilizes the altered embeddings. Our approach facilitates stable and adaptive clustering of the features. This results in a notable average improvement of 7\% points in zero-shot AUC scores across 40 classes in the MIMIC-CXR-JPG dataset from previous SOTA models.
Figures
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2406.04449 (2024) 1, 2
Bannur, S., Bouzid, K., Castro, D.C., Schwaighofer, A., Thieme, A., Bond-Taylor, S., Ilse, M., Pérez-García, F., Salvatelli, V., Sharma, H., et al.: Maira-2: Grounded radiology report generation. arXiv preprint arXiv:2406.04449 (2024) 1, 2
arXiv 2024
-
[2]
Boecking, B., et al.: Cxr-bert: Pretraining chest x-ray reports for multimodal align- ment.JournalofBiomedicalInformatics(2022),https://huggingface.co/microsoft/ BiomedVLP-CXR-BERT-general 2
work page 2022
-
[3]
Chen, K., Lei, W., Zhang, R., Zhao, S., shi Zheng, W., Wang, R.: Pcct: Progressive class-center triplet loss for imbalanced medical image classification (2022), https: //arxiv.org/abs/2207.04793 4
work page Pith review arXiv 2022
-
[4]
In: 34th British Machine Vision Conference 2023, BMVC 2023, Aberdeen, UK, November 20-24,
Delitzas, A., Parelli, M., Hars, N., et al., G.V.: Multi-clip: Contrastive vision- language pre-training for question answering tasks in 3d scenes. In: 34th British Machine Vision Conference 2023, BMVC 2023, Aberdeen, UK, November 20-24,
work page 2023
-
[5]
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T.: An image is worth 16x16 words: Transformers for image recognition at scale (2021), https://arxiv.org/abs/2010.11929 6
arXiv 2021
-
[6]
In: proceedings of Medical Image ComputingandComputerAssistedIntervention–MICCAI2024.vol.LNCS15012
Du, Y., Chang, B., Dvornek, N.C.: CLEFT. In: proceedings of Medical Image ComputingandComputerAssistedIntervention–MICCAI2024.vol.LNCS15012. Springer Nature Switzerland (October 2024) 2
work page 2024
-
[7]
Nature Biomedical Engi- neering pp
E, T., Talius, Patel, et al., P.: Expert-level detection of pathologies from unan- notated chest x-ray images via self-supervised learning. Nature Biomedical Engi- neering pp. 1399–1406 (2022). https://doi.org/10.1038/s41551-022-00936-9, https: //doi.org/10.1038/s41551-022-00936-9 6, 7
-
[8]
G, S.V., Ponraj, N., L, D.P.: Study on public chest x-ray data sets for lung dis- ease classification. In: 2021 3rd International Conference on Signal Processing and Communication(ICPSC).pp.54–58(2021).https://doi.org/10.1109/ICSPC51351. 2021.9451726 1
Show all 32 references
-
[9]
Holste, G., Wang, S., Jiang, Z., Shen, T.C., Shih, G., Summers, R.M., Peng, Y., Wang, Z.: Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study, p. 22–32. Springer Nature Switzerland (2022). https://doi.org/ 10.1007/978-3-031-17027-0_3, http://dx....
2022 doi
-
[10]
Medical Image Analysis (2021), https://arxiv.org/html/ 2312.07353v3 2
Huang, Z., et al.: Gloria. Medical Image Analysis (2021), https://arxiv.org/html/ 2312.07353v3 2
2021 arXiv
-
[11]
Johnson, A.E.W., Pollard, T.J., Greenbaum, N.R., Lungren, M.P., ying Deng, C., Peng, Y., Lu, Z., Mark, R.G., Berkowitz, S.J., Horng, S.: Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs (2019), https://arxiv.org/ abs/1901.07042 2, 5, 6
2019 arXiv
-
[12]
Communications for Statistical Applications and Methods 21 (03 2014)
Lee, H.Y., Park, H.J., Kim, H.M.: A clarification of the cauchy distribution. Communications for Statistical Applications and Methods 21 (03 2014). https: //doi.org/10.5351/CSAM.2014.21.2.183 4
2014 doi
-
[13]
Ley,C.,Neven,A.:Thevalueatthemodeinmultivariate tdistributions:acuriosity or not? (2014), https://arxiv.org/abs/1211.1174 7
2014 arXiv
-
[14]
Empirical Economics 58, 1461–1490 (2020)
Li, R., Nadarajah, S.: A review of student’s t distribution and its general- izations. Empirical Economics 58, 1461–1490 (2020). https://doi.org/10.1007/ s00181-018-1570-0, https://doi.org/10.1007/s00181-018-1570-0 2, 3
2020 doi
-
[15]
IEEE TMI (2023), https://www.researchgate.net/publication/374348635_ CXR-CLIP_Toward_Large_Scale_Chest_X-ray_Language-Image_ Pre-training 2 10 R
Li, Y., et al.: Clip for cxr: Fine-tuning clip for chest x-ray disease classification. IEEE TMI (2023), https://www.researchgate.net/publication/374348635_ CXR-CLIP_Toward_Large_Scale_Chest_X-ray_Language-Image_ Pre-training 2 10 R. Madhipati et al
2023
-
[16]
Messina,P.,Vidal,R.,Parra,D.,ÁlvaroSoto,Araujo,V.:Extractingandencoding: Leveraging large language models and medical knowledge to enhance radiological text representation (2024), https://arxiv.org/abs/2407.01948 4
2024 arXiv
-
[17]
Mu, N., Kirillov, A., Wagner, D., Xie, S.: Slip: Self-supervision meets language- image pre-training (2021), https://arxiv.org/abs/2112.12750 1, 6
2021 arXiv
-
[18]
Mukherjee, K., Khare, A., Verma, A.: A simple dynamic learning rate tuning al- gorithm for automated training of dnns (2019), https://arxiv.org/pdf/1910.11605 6
2019 arXiv
-
[19]
Nature Communications 13, 3848 (2022)
Park, S., Kim, G., Oh, Y.e.a.: Self-evolving vision transformer for chest x-ray diagnosis through knowledge distillation. Nature Communications 13, 3848 (2022). https://doi.org/10.1038/s41467-022-31514-x, https://doi.org/10. 1038/s41467-022-31514-x 2
2022 doi
-
[20]
Statistics and Computing10(4), 339–348 (2000) 3
Peel, D., McLachlan, G.J.: Robust mixture modelling using the t distribution. Statistics and Computing10(4), 339–348 (2000) 3
2000
-
[21]
https://doi.org/10
Peng, Y., Lin, M., Holste, G., Wang, S., Zhou, e.a.: Cxr-lt 2024: Long-tailed, multi- label, and zero- shot classification on chest x-rays (Apr 2024). https://doi.org/10. 5281/zenodo.10991413, https://doi.org/10.5281/zenodo.10991413 2, 5
2024 doi
-
[23]
Radford,A.,Kim,J.W.,Hallacy,C.,Ramesh,A.,etal.,G.G.:Learningtransferable visual models from natural language supervision (2021), https://arxiv.org/abs/ 2103.00020 1, 2, 4, 7
2021 arXiv
-
[24]
Seputis, D., Mihailov, S., Chatterjee, S., Xiao, Z.: Multi-modal adapter for vision- language models (2024), https://arxiv.org/abs/2409.02958 1
2024 arXiv
-
[25]
In: WACV
Shentu, J., Al Moubayed, N.: Cxr-irgen: An integrated vision and language model for the generation of clinically accurate chest x-ray image-report pairs. In: WACV. pp. 5200–5209 (2024). https://doi.org/10.1109/WACV57701.2024.00513 2
2024
-
[26]
Speets, A.M., van der Graaf, Y., Hoes, A.W., et al.: Chest radiography in general practice: indications, diagnostic yield and consequences for patient management (2006), https://pubmed.ncbi.nlm.nih.gov/16882374/ 1
2006
-
[27]
Wan, H., Wang, H., Scotney, B., Liu, J.: A novel gaussian mixture model for classification. In: SMC. pp. 3298–3303 (2019). https://doi.org/10.1109/SMC.2019. 8914215 2, 3, 4
2019 doi
-
[28]
Journal of Medical Internet Research (2023), https://hai.stanford.edu/news/ ai-researchers-tap-medical-twitter-create-powerful-new-analysis-tool 2
Wang, X., et al.: Clip on medical twitter: Leveraging social media for disease detec- tion. Journal of Medical Internet Research (2023), https://hai.stanford.edu/news/ ai-researchers-tap-medical-twitter-create-powerful-new-analysis-tool 2
2023
-
[30]
Wu, C., Zhang, X., Zhang, Y., Wang, Y., Xie, W.: Medklip: Medical knowledge enhanced language-image pre-training in radiology (2023), https://arxiv.org/abs/ 2301.02228 1, 2, 6
2023 arXiv
-
[31]
In: Greenspan, H.e.a
You, K.e.a.: Cxr-clip: Toward large scale chest x-ray language-image pre-training. In: Greenspan, H.e.a. (ed.) MICCAI 2023. Lecture Notes in Computer Science, vol. 14221, pp. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43895-0_ 10, https://doi.org/10.1007/978-3-03...
2023 doi
-
[32]
Nature Communications 14, 4542 (2023)
Zhang, X., Wu, C., Zhang, Y., Xie, W., Wang, Y.: Knowledge-enhanced visual-language pre-training on chest radiology images. Nature Communications 14, 4542 (2023). https://doi.org/10.1038/s41467-023-40260-7, https://doi.org/10. 1038/s41467-023-40260-7 2
2023 doi
-
[33]
NeurIPS (2020), https://arxiv.org/pdf/2210.10163 2
Zhang, Y., et al.: Convirt:. NeurIPS (2020), https://arxiv.org/pdf/2210.10163 2
2020 arXiv
-
[2023]
BMVA (2023), https://papers.bmvc2023.org/0748.pdf 1
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.