Pith. sign in

REVIEW 4 major objections 5 minor 17 references

A Vision-Language Model for Focal Liver Lesion Classification

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read A text-aligned vision-language model beats standard CLIP and MedCLIP on focal liver lesion CT classification.

desk verdict A modest, honest CLIP-style application to liver lesion classification whose internal comparisons are useful, but whose headline outperformance claim over CLIP/MedCLIP rests on an unstated and likely zero-shot baseline protocol. read the letter →

arxiv 2505.03350 v1 pith:XBII4U4Y submitted 2025-05-06 cs.CV

classification cs.CV
keywords focalliverlesionsvision-languagemodelmultimodallearningmulti-phaseCTcross-entropyalignmentCLIPmedicalimageclassificationsmall-sample
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Liver-VLM, a vision-language model that classifies focal liver lesions from three-phase CT slices by aligning image features with class-specific text embeddings. It claims this text-guided alignment, trained with cross-entropy loss on cosine similarities, beats standard CLIP and MedCLIP on the MPCT-FLLs dataset, which contains only 489 annotated slices. The best variant, with an ImageNet-pretrained ResNet18 encoder and a frozen BERT text encoder, reaches $79.27\%$ average accuracy and $0.91$ AUC. If the claim holds, it offers a practical route to useful medical classification where large annotated datasets are unavailable.

What carries the argument

The load-bearing mechanism is cross-entropy alignment between image and text embeddings. Class names are expanded to full labels and placed in the prompt template 'a CT scan of tumors {label}', the frozen BERT text encoder turns each prompt into a class embedding, the image encoder output is projected by a fully connected layer to the same dimension, and the softmax of cosine similarities between the image and all class embeddings is trained with cross-entropy. The frozen text encoder and prompt template are what carry the class prior; the only trainable text-side component is the projection layer, so the added text guidance costs no extra computation at inference time.

What would settle it

Fine-tune CLIP and MedCLIP on the same 3-fold MPCT-FLLs splits, with an ImageNet-initialized ResNet backbone, the same optimizer, the same 200 epochs, and the same 'a CT scan of tumors {label}' prompt template. If either fine-tuned baseline reaches or exceeds $79.27\%$ average accuracy or $0.91$ AUC, the paper's central claim that Liver-VLM outperforms them is falsified.

Watch

Extended reading notes

Core claim

Liver-VLM is a CLIP-inspired framework in which a trainable image encoder (ResNet50 or ResNet18) embeds each three-phase CT slice and a frozen BERT text encoder embeds the class prompt 'a CT scan of tumors {label}'; a trainable projection layer brings the two modalities into a shared space, and cross-entropy loss over pairwise cosine similarities aligns each image with its true class text. The paper's central claim is that this class-level text alignment yields more discriminative features for focal liver lesions than standard CLIP or MedCLIP under annotation-scarce conditions. On the MPCT-FLLs dataset, CLIP and MedCLIP collapse to two classes and hover near chance, while every Liver-VLM variant classifies all four lesion types; the best configuration reaches $79.27\pm3.06\%$ average accuracy and $0.91\pm0.03$ AUC. The authors also claim that a lighter ResNet18 backbone performs better than ResNet50 in this data-constrained setting, and they state the resulting accuracy is still insufficient for reliable clinical application.

Load-bearing premise

The load-bearing premise is that the CLIP and MedCLIP baselines were evaluated under the same training protocol as Liver-VLM; the paper does not state whether those baselines were fine-tuned or used off the shelf, so the comparison may not be apples to apples.

Editorial extensions

If this is right

  • Class-level text conditioning can be applied to other small-sample medical imaging tasks with few labels.
  • A lightweight ResNet18 backbone should be preferred over ResNet50 when only a few hundred training slices are available.
  • Training from scratch with text alignment already outperforms the CLIP and MedCLIP baselines, so the text signal itself, not only ImageNet pretraining, drives the improvement.
  • The best model's $0.91$ AUC suggests acceptable ranking for triage, while its $79.27\%$ accuracy is below what the paper considers clinically reliable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reported comparison may be unfair if CLIP and MedCLIP were not fine-tuned; a same-protocol fine-tuned baseline is the decisive missing experiment.
  • Editorial inference: changing the prompt template could shift results, so the method's sensitivity to wording should be tested directly.
  • Editorial inference: because the 489 slices come from a single institution, external multi-center data are needed to test whether text-aligned embeddings generalize.
  • Editorial inference: adding a contrastive term over image pairs to the cross-entropy alignment might further separate classes whose text descriptions differ only slightly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Liver-VLM, a CLIP-inspired model for classifying focal liver lesions (FLLs) in multi-phase CT images. The model uses a ResNet image encoder and a frozen BERT text encoder with class-level prompt embeddings, trained with a cross-entropy loss over cosine similarities. Experiments on the MPCT-FLLs dataset (489 slices, four lesion types) compare two variants (train from scratch and ImageNet fine-tuning) with ResNet50 and ResNet18 backbones against CLIP and MedCLIP baselines. The authors report that Liver-VLM outperforms CLIP and MedCLIP in accuracy and AUC, and that the lightweight ResNet18 backbone performs better than ResNet50 under limited-data conditions.

Significance. If the claims were fully supported, the work would offer a modest contribution to low-data medical image classification by showing that aligning image features with frozen text class prototypes can provide a strong inductive bias. The paper's internal comparisons (Model 1 vs. Model 2, ResNet50 vs. ResNet18) are clearly reported and internally consistent. However, the central external comparison against CLIP and MedCLIP is currently not credible because the baseline protocol is undisclosed and the reported baseline performances are characteristic of zero-shot evaluation rather than equivalent supervised training. The absence of an image-only control further prevents attribution of the gains to the text modality. The paper therefore does not yet substantiate its main claim.

major comments (4)
  1. [§4.3, Table 3] The protocols for the CLIP and MedCLIP baselines are not specified. The authors state that a CLIP model with ResNet50 was used as a baseline and that MedCLIP was evaluated on the same dataset, but they do not state whether these models were fine-tuned on the MPCT-FLLs training folds, used in a zero-shot fashion, or adapted with any other procedure. The reported results—CLIP at 32.80% accuracy with 0% on FNH and HEM, and MedCLIP at 30.73% with 0% on FNH and HCC—are exactly what one would expect from frozen, off-the-shelf models on an out-of-domain CT dataset. Because the Liver-VLM models are trained for 200 epochs with labels, the comparison conflates the training procedure with the model architecture. The abstract claim that Liver-VLM 'outperforms both the standard CLIP and MedCLIP models' is therefore not supported unless the baselines received equivalent supervised fine-tuning under the same splits and hyperparameters.
  2. [§3, §4.3 (Tables 3 and 4)] No image-only baseline is included. The proposed method aligns image embeddings with text embeddings and optimizes with cross-entropy, but the paper never trains a standard ResNet18 or ResNet50 classifier with the same training procedure, data splits, and a simple linear head (or the same projection layer without text) to measure the contribution of the text branch. Without this control, the reported improvements over CLIP/MedCLIP may be entirely due to the ResNet backbone and the supervised training scheme, and the paper's central claim that the vision-language alignment improves small-sample classification is not established.
  3. [§4.1, Table 1] The data split protocol is ambiguous. The text says 'In each fold, one group was designated as the test set, while the other two were used for training,' but Table 1 lists a third 'Val' subset. The authors should clarify whether the validation group is used for model selection/early stopping, and if so, how it is incorporated into the 3-fold cross-validation. This matters for the reproducibility and fairness of all reported comparisons.
  4. [§4.3, Tables 3 and 4] The claim that ResNet18 'enhances classification performance' over ResNet50 is based on differences between averages from only three folds, with standard deviations that overlap (e.g., ResNet50 Model 2: 74.08±4.33; ResNet18 Model 2: 79.27±3.06). The authors should report per-fold results and perform a paired statistical test (e.g., Wilcoxon signed-rank test) or otherwise demonstrate that the observed differences are not due to fold variance.
minor comments (5)
  1. [Abstract and §1] There is a typo in the abstract: 'pro-pose' should be 'propose', and 'classifiers' in 'which classifiers image' should be 'classifies'. The grammar in several places ('However, its reliance...', 'a detailed description of the proposed approach .') needs editing.
  2. [§3.4] The prompt template 'a CT scan of tumors {label}' is described, but the exact prompt strings used for each class are not listed. Since prompt wording can materially affect zero-shot or text-encoder behavior, the authors should include the final prompts for Cyst, FNH, HCC, and HEM.
  3. [§4.2, Table 2] Table 2 is titled 'Computation Environment' but lists no hyperparameters such as learning rate schedule, weight decay, or the number of warm-up steps. The training setup described in the text (200 epochs, batch size 32, learning rate 0.01, AdamW) is helpful, but additional details would improve reproducibility.
  4. [§4.3] The standard deviations for per-class accuracies are large (e.g., HEM accuracy 63.15±25.12 for ResNet50 Model 2), yet the text discusses these differences as if they were meaningful. The authors should acknowledge the high variance and avoid over-interpreting per-class changes without statistical support.
  5. [References] Reference [13] for MedCLIP is cited as a MICCAI 2022 paper, but the MedCLIP model has multiple versions (e.g., with ViT or CNN encoders). The authors should specify which MedCLIP checkpoint and variant was used and how it was obtained.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Liver-VLM is supervised classification with fixed class-prototype text embeddings; its claimed gains are empirical and not self-derived.

full rationale

The paper's central claim is that Liver-VLM, trained with cross-entropy loss to align image embeddings with frozen BERT text embeddings, achieves higher accuracy and AUC than CLIP and MedCLIP on the MPCT-FLLs dataset. Nothing in the method defines the predicted label in terms of the evaluation metric, nor fits a parameter and then renames it as a prediction. The text prompts are fixed class descriptions ('a CT scan of tumors {label}'), and the image encoder is trained on labeled ROI slices; this is ordinary supervised learning with fixed class prototypes. The paper's own equation (Eq. 1) is the standard cross-entropy loss, not an identity that assumes the reported result. Self-citations [4-6] appear only as background about prior pretraining work and are not used to justify the proposed model's outcome; no uniqueness theorem or ansatz is imported from the authors' prior work. The most serious concern is experimental rather than circular: Section 4.3 reports CLIP and MedCLIP baselines without stating whether they were fine-tuned under the same protocol, and their near-chance, degenerate per-class results (e.g., FNH and HEM at 0% for CLIP) suggest zero-shot evaluation. That is a validity and fair-comparison issue, not a circularity, because even if the baselines were zero-shot, Liver-VLM's result is not derived from the baselines by construction; it is an independent empirical measurement. The paper also explicitly acknowledges that accuracy remains insufficient for clinical application, which further indicates the claims are empirical rather than tautological. No load-bearing step reduces to its own input, so the circularity score is 0.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities or theoretical constructs. Its central claim rests on the hand-chosen prompt and hyperparameters, the correctness of the labels, and the assumption that the evaluation is fair. The network weights are fitted to the training data, and the baselines' training status is unstated, which is the largest unexamined dependency.

free parameters (7)
  • Image encoder and projection weights = ResNet50 or ResNet18 plus FC_T, trained on MPCT-FLLs training folds
    All classification accuracy numbers depend on these fitted weights; they are the primary fitted quantities in the supervised training.
  • Learning rate = 0.01
    Hand-chosen for AdamW optimizer; no ablation shown for sensitivity.
  • Batch size = 32
    Chosen for training; no ablation.
  • Training epochs = 200
    Fixed training length; no early stopping described.
  • Input image resize = 128x128
    2D ROI slices are resized from 512x512; no ablation of resolution.
  • Prompt template = "a CT scan of tumors {label}"
    Hand-crafted prompt; the text embeddings and resulting classifier prototypes depend on it.
  • Trainable projection layer = unspecified output dimension
    An FC layer projects image embeddings to the BERT embedding dimension; its size is not stated.
assumptions (5)
  • domain assumption Radiologist annotations on the MPCT-FLLs dataset are correct and consistent.
    All training and evaluation depends on ground-truth lesion labels; no inter-observer variability is reported.
  • domain assumption Representing the three CT phases as a 3-channel RGB-like input preserves diagnostically relevant information.
    The method treats NC, ART, and PV slices as channels of one image; this is a modeling choice stated in Section 4.1.
  • domain assumption Frozen BERT embeddings of the prompt provide meaningful class targets for alignment.
    The text encoder is never fine-tuned; the success of the method depends on these fixed text embeddings being separable and relevant to CT image features.
  • domain assumption The cross-validation split separates patients or cases so that slices from the same volume do not appear in both training and test folds.
    The paper reports case counts per group in Table 1 but does not explicitly state that no volume overlaps folds; leakage across slices of the same lesion would inflate accuracy.
  • domain assumption Pretrained ImageNet weights are a beneficial initialization for CT images.
    Model 2 uses ImageNet pretraining; if this transfer assumption fails, the fine-tuning variant's results would differ.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Vision-Language Model for Focal Liver Lesion Classification." pith.science (2026). https://pith.science/paper/XBII4U4Y

@misc{pith2026250503350,
  author       = {Pith},
  title        = {Pith review of: A Vision-Language Model for Focal Liver Lesion Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XBII4U4Y}},
  note         = {Machine review of arXiv:2505.03350}
}
read the original abstract

Accurate classification of focal liver lesions is crucial for diagnosis and treatment in hepatology. However, traditional supervised deep learning models depend on large-scale annotated datasets, which are often limited in medical imaging. Recently, Vision-Language models (VLMs) such as Contrastive Language-Image Pre-training model (CLIP) has been applied to image classifications. Compared to the conventional convolutional neural network (CNN), which classifiers image based on visual information only, VLM leverages multimodal learning with text and images, allowing it to learn effectively even with a limited amount of labeled data. Inspired by CLIP, we pro-pose a Liver-VLM, a model specifically designed for focal liver lesions (FLLs) classification. First, Liver-VLM incorporates class information into the text encoder without introducing additional inference overhead. Second, by calculating the pairwise cosine similarities between image and text embeddings and optimizing the model with a cross-entropy loss, Liver-VLM ef-fectively aligns image features with class-level text features. Experimental results on MPCT-FLLs dataset demonstrate that the Liver-VLM model out-performs both the standard CLIP and MedCLIP models in terms of accuracy and Area Under the Curve (AUC). Further analysis shows that using a lightweight ResNet18 backbone enhances classification performance, particularly under data-constrained conditions.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 14 canonical work pages

  1. [1]

    Radiology, Vol.286, pp.887– 896 (2017)

    Yasaka, K., et al.: Deep learning with convolutional neural network for differentiation of liver masses at dynamic contrast -enhanced CT: A preliminary study. Radiology, Vol.286, pp.887– 896 (2017)

  2. [2]

    In: Frangi A., Schnabel J., Davat- zikos C., Alberola-Ló pez C., Fichtinger G

    Liang, D., et al.: Combining Convolutional and Recurrent Neural Networks for Classifica- tion of Focal Liver Lesions in Multi -Phase CT Imaging. In: Frangi A., Schnabel J., Davat- zikos C., Alberola-Ló pez C., Fichtinger G. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, LNCS, vol. 7951, pp.666-675. Springer (2018)

  3. [3]

    Wang, W., et al.: Classification of Focal Liver Lesions Using Deep Learning with Fine - tuning, In: Proceedings of Digital Medicine and Image Processing (DMIP2018), pp.56 -60, (2018)

  4. [4]

    al.: Case Discrimination: Self-supervised Feature Learning for the classifica- tion of Focal Liver Lesions,” in Chen, Y

    Dong, H., et. al.: Case Discrimination: Self-supervised Feature Learning for the classifica- tion of Focal Liver Lesions,” in Chen, Y. -W. et al. (eds.), Innovation in Medicine and Healthcare, Smart Innovation, Systems and Technologies (Proc. of InMed2021), pp 241 - 249, Springer (2021)

  5. [5]

    -W.: Deep Neural Network - Based Classification of Focal Liver Lesions Using Phase-Shuffle Prediction Pre-training

    Song, J., Dong, H., Chen, Y., Lin, L., Hu, H., and Chen, Y. -W.: Deep Neural Network - Based Classification of Focal Liver Lesions Using Phase-Shuffle Prediction Pre-training. In: Chen, Y.-W., et al. (eds.) Innovation in Medicine and Healthcare. KES InMed 2023. Smart Innovation, Systems and Technologies, vol. 357, pp. 235 –243. Springer (2023) doi:10.1007...

  6. [6]

    -W.: Early Recur- rence Prediction of Hepatocellular Carcinoma Usin g Deep Learning Frameworks with Multi-Task Pre-Training

    Song, J., Dong, H., Chen, Y., Zhang, X., Zhan, G., Jain, R.K., Chen, Y. -W.: Early Recur- rence Prediction of Hepatocellular Carcinoma Usin g Deep Learning Frameworks with Multi-Task Pre-Training. Information 15(8), 493 (2024). doi:10.3390/info15080493

  7. [7]

    In: CVPR 2021, pp

    Desai, K., Johnson, J.: VirTex: Learning Visual Representations from Textual Annotations. In: CVPR 2021, pp. 11162–11172. IEEE, Nashville (2021)

  8. [8]

    In: ACM MM 2020, pp

    Sariyildiz, B., Nazarieh, S., Ricci, E.: ICMLM: Image Conditioned Masked Language Mod- eling. In: ACM MM 2020, pp. 2590–2598. ACM, Seattle (2020)

Show all 17 references
  1. [9]

    In: arXiv preprint arXiv:2010.00747 (2020)

    Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive Learning of Medical Visual Representations from Paired Images and Text. In: arXiv preprint arXiv:2010.00747 (2020)

  2. [10]

    In: ICML 2021, pp

    Radford, A., Kim, J., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning Transferable Visual Models From Natural Language Supervision. In: ICML 2021, pp. 8748–8763. PMLR, Virtual (2021)

  3. [11]

    R., Gotmare, A., Joty, S., Xiong, C., Hoi, S

    Li, J., Selvaraju, R. R., Gotmare, A., Joty, S., Xiong, C., Hoi, S. C.: BLIP: Bootstrapped Language-Image Pre-training for Unified Vision-Language Understanding and Generation. arXiv preprint arXiv:2201.12086 (2022)

  4. [12]

    arXiv preprint arXiv:2304.08485 (2023)

    Liu, H., Peng, H., Yu, Z., Ma, X., Wang, J., Ma, H., Wang, Y., Wu, J., Xie, S.: Visual Instruction Tuning. arXiv preprint arXiv:2304.08485 (2023)

  5. [13]

    In: MICCAI 2022, LNCS, vol

    Wang, Z., Yang, J., Wang, D., Xu, Y., Bai, J., Zhou, S.K.: MedCLIP: Contrastive Learning from Unpaired Medical Images and Text. In: MICCAI 2022, LNCS, vol. 13433, pp. 40–50. Springer, Singapore (2022)

  6. [14]

    In: CVPR 2016, pp

    He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: CVPR 2016, pp. 770–778. IEEE, Las Vegas (2016)

  7. [15]

    In: NAACL -HLT 2019, pp

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: NAACL -HLT 2019, pp. 4171–4186. ACL, Minneapolis (2019)

  8. [16]

    In: Medical Physics, vol

    Xu, Y., et al.: PA-ResSeg: A Phase Attention Residual Network for Liver Tumor Segmen- tation from Multi -phase CT Images. In: Medical Physics, vol. 48, no. 7, pp. 3752 –3766. AAPM, (2021)

  9. [17]

    In: ICLR 2019

    Loshchilov, I., Hutter, F.: Decoupled Weight Decay Regularization. In: ICLR 2019. Open- Review, New Orleans (2019)

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.