Pith. sign in

REVIEW 5 major objections 5 minor 20 references

ENCLIP: Ensembling and Clustering-Based Contrastive Language-Image Pretraining for Fashion Multimodal Search with Limited Data and Low-Quality Images

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ENCLIP claims that ensembling five CLIP models fine-tuned for different epoch counts improves fashion text-to-image search on small, low-quality datasets beyond pretrained CLIP and FashionCLIP.

desk verdict Useful problem and a plausible ensemble idea, but Algorithm 1 has no text input and there is no fine-tuned baseline, so the reported text-to-image numbers do not stand. read the letter →

arxiv 2411.16096 v1 pith:3BOAL4Z4 submitted 2024-11-25 cs.CV cs.AIcs.MM

Prithviraj Purushottam Naik , Rohit Agarwal This is my paper · ORCID
classification cs.CVcs.AIcs.MM
keywords multimodalsearchinformationretrievalfashione-commerceensemblingclusteringCLIPcontrastivelearninglow-resolutionimages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a single fine-tuned CLIP model is not the best way to adapt CLIP to small, low-quality fashion image collections, and that ensembling several fine-tuned CLIP models can do better. It proposes ENCLIP, which trains five CLIP models for 10, 30, 50, 80, and 100 epochs, encodes the dataset images with each, and then selects retrieval results by weighted frequency scores plus K-means cluster-head ranking. On the Fashion Product Images (Small) dataset, ENCLIP reports higher mean average precision at 10 (mAP@10) than both pretrained CLIP and FashionCLIP for most of the tested query categories. If the result holds, it gives a practical recipe for squeezing better text-to-image search out of CLIP when training data are scarce and images are low resolution, as is common in fashion e-commerce.

What carries the argument

The central object is the ENCLIP selection algorithm (Algorithm 1), a weighted-voting and cluster-head ranking procedure. Each fine-tuned CLIP model outputs a latent image embedding; occurrences of an image across models define a frequency, and the weighted score is $\sum_{n=0}^{z-1} 0.1(2n)\times\text{occurrence}$, which gives later training stages greater influence. K-means is applied to the t-SNE-reduced latent space to define clusters, and the most frequent image in each cluster acts as the head image whose cluster is then ranked by frequency and weighted score. The mechanism's job is to let early-epoch and late-epoch checkpoints complement each other: early checkpoints generalize on scarce data, late checkpoints capture finer distinctions, and the cluster step suppresses images that only one model accidentally retrieved.

What would settle it

Re-run ENCLIP exactly as described on the same dataset with the same 100 queries and compare the resulting mAP@10 values with Table 3; if the procedure cannot accept a text query or the numbers do not match, the central claim is falsified. A smaller check: the stated score gives zero weight to the first fine-tuned model, so removing it should not change the ranking if the formula is applied literally.

Watch

Extended reading notes

Core claim

ENCLIP is an ensemble-and-cluster method for fine-tuning CLIP in a data-poor, low-image-quality setting. Five CLIP models are initialized identically and trained for different numbers of epochs, and each model's image outputs are treated as votes. An image's frequency across model outputs and a weighted score $0.1 \times (2n)$ per occurrence at model stage $n$ determine priority, and K-means clustering in a t-SNE-reduced latent space selects head clusters from which the top $N$ images are ranked. The paper reports that this combined ranking gives mAP@10 values above pretrained CLIP in all ten category-subcategory cells and above FashionCLIP in eight of ten cells, with the largest gains on ethnic wear for men and women's footwear. The intended message is that early-epoch models generalize and late-epoch models are precise, so their ensemble plus clustering is a better fit for limited-data fashion search than any single checkpoint.

Load-bearing premise

The paper's central assumption is that its described ENCLIP procedure is what produced the reported retrieval numbers; the listed algorithm takes no text query, so the step that connects a user's text to the image selection is unstated.

Editorial extensions

If this is right

  • The reported mAP@10 places ENCLIP above pretrained CLIP in all ten category-subcategory cells and above FashionCLIP in eight of ten cells for the 100 queries tested.
  • The approach achieves these numbers with roughly 35k low-resolution training images, about 20 times fewer than the high-resolution set used to train FashionCLIP, so the ensemble-and-cluster recipe is presented as a substitute for a large clean dataset.
  • The authors report the largest relative gains in categories where pretrained CLIP is weakest, such as ethnic wear for men, so ensembling is claimed to recover signal in underrepresented fashion segments.
  • Because the method only adds standard CLIP fine-tuning runs, t-SNE projection, and K-means, it can be applied to an existing small catalog without collecting new images or changing the model architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader of Algorithm 1 will notice that its inputs are image outputs and a number $N$, with no text query; since the paper reports text-to-image mAP@10, there must be an unreported step that embeds the query and combines it with the image-based selection, and that step should be made explicit before the results are reproduced.
  • The weighting formula assigns zero weight to the first model's occurrences, so one testable implication is that dropping the epoch-10 checkpoint should not change the ranking; if it does, the formula in the paper is not the one actually evaluated.
  • The same ensemble-and-cluster procedure could be applied to other contrastive image-text models and to other small, noisy image domains once the query-fusion step is specified.
  • The evaluation uses only 10 queries per category, so a larger query set would test whether the reported margins are stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes ENCLIP, an ensemble of five CLIP models fine-tuned for 10, 30, 50, 80, and 100 epochs on the Fashion Product Images (Small) dataset, combined through a weighted frequency score and K-means cluster-head selection. The authors claim that ENCLIP improves text-to-image fashion search over pretrained CLIP and FashionCLIP, as measured by mAP@10 across ten fashion categories. The manuscript includes qualitative retrieval examples, t-SNE visualizations, and a comparison table; it does not release code or specify how text queries enter the proposed algorithm.

Significance. If the central claim were supported, ENCLIP would offer a practical, low-resource recipe for adapting CLIP to small and low-quality fashion collections, using about 35k low-resolution images instead of the roughly 700k high-resolution images used by FashionCLIP. The paper also clearly identifies a real domain problem and reports results on a public dataset. However, the evidence as presented does not establish the claim: Algorithm 1 does not take a text query as input, the quantitative comparison does not include a single fine-tuned CLIP baseline, the relevance labels used for mAP are undefined, and key hyperparameters appear to be selected on the evaluation queries. These gaps concern the central claim and currently prevent the paper from being a reproducible or convincing empirical study.

major comments (5)
  1. [Section 4.4, Algorithm 1] Algorithm 1 is presented as the ENCLIP retrieval method, but its inputs are only image outputs M = {m1,...,mz} and an integer N; no text query is provided. The paper reports text-to-image retrieval results for queries such as "Give me polo neck t-shirt for men" (Tables 1 and 2 and Figures 5 and 6), yet it never specifies how the query string is embedded, how its embedding is compared with image embeddings, or how the frequency/weighted-score/clustering steps interact with that comparison. As written, the algorithm cannot produce the reported retrieval numbers, so the central claim lacks a defined mechanism.
  2. [Table 3 and Section 5] The only quantitative comparison is against zero-shot pretrained CLIP and zero-shot FashionCLIP, not against a single CLIP model fine-tuned on the same training data. Because ENCLIP is fine-tuned and the baselines are zero-shot, the reported gains could be due entirely to fine-tuning rather than to the ensemble or clustering strategy. The paper needs matched baselines such as individually fine-tuned CLIP models at each epoch count, and ideally an ablated ENCLIP without clustering, to isolate the contribution of the proposed components.
  3. [Equation (6) and Section 5] The mAP computation relies on rel(k), the relevance function, but the paper never defines relevance labels for the Fashion Product Images (Small) dataset or describes how ground truth is constructed for the 100 evaluation queries. Without a definition of what makes a retrieved image relevant to a query, the mAP numbers in Table 3 are not interpretable or reproducible. This is a load-bearing omission for an empirical retrieval paper.
  4. [Section 4.4 and Section 5] The cluster count k in [4,6] and the ranking criterion "frequency and weighted_score" are chosen after observing performance on the evaluation queries ("The K value between the range of 4 and 6 gave the best results" and "Ranking based on frequency and weighted_score yielded the best results"). No validation set or error bars are reported. This post-hoc selection on the test queries makes the reported mAP values optimistic and prevents a fair comparison with baselines that do not benefit from such selection.
  5. [Section 4.3, Equation (1)] The training loss is described as the average of a categorical cross-entropy image loss and a categorical cross-entropy text loss, rather than the contrastive image-text loss used by CLIP. This is inconsistent with the paper's claim of fine-tuning CLIP and raises the question of what objective the individual models actually optimize. The authors should clarify whether they replace CLIP's contrastive objective and, if so, how the learned representations retain the cross-modal alignment needed for text-to-image search.
minor comments (5)
  1. [Equations (2) and (3)] Equation (2) in Section 4.4 and Equation (3) in Algorithm 1 are identical and both numbered; one of the two should be removed to avoid confusion.
  2. [Throughout] There are several typographical issues, including "In 2023 Min Wang etal." missing a space, "FashionCLIP, being a CLIP-like model,require" missing an 's', and inconsistent spacing around commas and parentheses.
  3. [Figures 5 and 6] The captions say the t-SNE plots and K-means results are "obtained by the query", but Algorithm 1 does not use a text query; the figures should either describe the actual selection process or be updated after the algorithm is revised to include query input.
  4. [Section 4.4] Clustering is applied to t-SNE-transformed features, but t-SNE is stochastic and primarily preserves local neighborhoods; the authors should justify why clustering in t-SNE space is preferable to clustering in the original 512-dimensional latent space.
  5. [Table 3] The table reports only point estimates of mAP@10 without standard deviations or per-query breakdowns; at minimum, the paper should report error bars over multiple runs or query subsets, especially because the ensemble contains stochastic components.

Circularity Check

1 steps flagged · score 5.0 of 10

ENCLIP's reported mAP is partly a post-hoc fit: the final ranking rule was chosen on the same 100 test queries used to report Table 3, and Algorithm 1 as written takes no text query.

  1. fitted input called prediction [Section 5, 'Evaluation Results and Discussion' (ranking-method paragraph after Fig. 6; same 100 queries feed Table 3)]
    "In Table. 3 results, 10 queries for each category to calculate Mean Average Precision is considered. So, a total of 100 queries are considered for evaluation. ... The evaluation and ranking of images have been conducted using the following methods: ... Sorting in descending order based on frequency and weighted_score. Ranking based on frequency and weighted_score yielded the best results in the findings."

    The final ENCLIP ranking rule is chosen by comparing three scoring rules on the same evaluation queries used for the headline mAP@10 in Table 3, and keeping the one that 'yielded the best results.' The reported ENCLIP column is therefore the output of a test-set selection loop: the ranking criterion is fitted to the 100 queries and then presented as part of the method's predicted advantage. The paper reports no validation split or error bars, so the outperformance claim is not independent of the selection procedure.

full rationale

The main circular element is confined to the ranking-rule selection in Section 5: the same 100 queries used to compute the Table 3 mAP numbers were used to pick 'frequency and weighted_score' as the ranking rule, so part of ENCLIP's reported gain is a post-hoc fit rather than an out-of-sample prediction. This is a genuine but limited circularity; the underlying comparison is an empirical benchmark, not a derivation from a self-citation or a uniqueness theorem. There are no load-bearing self-citations: FashionCLIP and pretrained CLIP are used as external baselines, not as premises that force the ENCLIP result. Separately, Algorithm 1's stated inputs are only image outputs and N, with no text-query input, while Figure 4 presents it as the text-to-image retrieval algorithm and Table 3 reports text-to-image mAP@10; this means the reported retrieval numbers cannot be reproduced from the method as written, a missing-support issue that further weakens the claim but is not itself an equivalence-by-construction. The conclusion's own limitation note — that ENCLIP 'may not be well-suited for fine-grained querying' — is consistent with this assessment. Overall score 5: one fitted component contaminates the headline comparison, but the result is not forced by definition or by self-citation.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method rests on several unstated assumptions about loss function validity, t-SNE distance semantics, and cross-model comparability, plus multiple hand-chosen free parameters that were tuned on the evaluation queries.

free parameters (5)
  • Ensemble weight multiplier = 0.1 x 2^n for n=0..4, i.e., 0.1, 0.2, 0.4, 0.8, 1.6
    Introduced in Equation 2/3 without derivation; weights are hand-picked to emphasize later epochs and are not ablated on a held-out set.
  • Number of fine-tuned models z = 5
    Chosen for demonstration; no analysis of how z affects performance.
  • Epoch schedule = 10, 30, 50, 80, 100
    Arbitrary set of training lengths; selected without justification.
  • Cluster count k = between 4 and 6
    Selected via elbow and silhouette method, but the exact k and its effect on ranking are not reported.
  • Ranking criterion = sort by frequency then weighted_score
    Chosen because it 'yielded the best results'; this is post-hoc selection on the test data.
assumptions (3)
  • domain assumption Categorical cross-entropy loss on image and text is a valid way to fine-tune CLIP for multimodal retrieval.
    Section 4.3 replaces CLIP's contrastive loss with cross-entropy, but the paper does not justify how this preserves cross-modal alignment.
  • domain assumption t-SNE projections preserve semantic cluster structure well enough for K-means to identify relevant groups.
    Section 4.4: K-means is applied to 2D t-SNE coordinates; this assumes distances in the projection reflect semantic similarity.
  • domain assumption The latent spaces of the five fine-tuned models are directly comparable, so occurrence counts across them are meaningful.
    Section 4.4: frequency and weighted scores are computed across model outputs without alignment or normalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ENCLIP: Ensembling and Clustering-Based Contrastive Language-Image Pretraining for Fashion Multimodal Search with Limited Data and Low-Quality Images." pith.science (2026). https://pith.science/paper/3BOAL4Z4

@misc{pith2026241116096,
  author       = {Pith},
  title        = {Pith review of: ENCLIP: Ensembling and Clustering-Based Contrastive Language-Image Pretraining for Fashion Multimodal Search with Limited Data and Low-Quality Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3BOAL4Z4}},
  note         = {Machine review of arXiv:2411.16096}
}
read the original abstract

Multimodal search has revolutionized the fashion industry, providing a seamless and intuitive way for users to discover and explore fashion items. Based on their preferences, style, or specific attributes, users can search for products by combining text and image information. Text-to-image searches enable users to find visually similar items or describe products using natural language. This paper presents an innovative approach called ENCLIP, for enhancing the performance of the Contrastive Language-Image Pretraining (CLIP) model, specifically in Multimodal Search targeted towards the domain of fashion intelligence. This method focuses on addressing the challenges posed by limited data availability and low-quality images. This paper proposes an algorithm that involves training and ensembling multiple instances of the CLIP model, and leveraging clustering techniques to group similar images together. The experimental findings presented in this study provide evidence of the effectiveness of the methodology. This approach unlocks the potential of CLIP in the domain of fashion intelligence, where data scarcity and image quality issues are prevalent. Overall, the ENCLIP method represents a valuable contribution to the field of fashion intelligence and provides a practical solution for optimizing the CLIP model in scenarios with limited data and low-quality images.

Figures

Figures reproduced from arXiv: 2411.16096 by the authors.

Figure 1
Figure 1. Contrastive pretraining with CLIP 4 Methodology 4.1 Dataset This dataset, fashion-product-images-small [1], provided by HuggingFace, is used in this study. It consists of around 44072 60x80 pixel resolution images with details and descriptions about the image in respective columns. It has 7 mas￾ter categories(Apparel, Accessories, Footwear, Personal Care, Free items, Home, Sporting Goods), 45 subcategories [PITH_FU… view at source ↗
Figure 2
Figure 2. Fine-tuning CLIP [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Storing the image embedding and texual embedding in vector database [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Process of retrieving the results from vector database using ENCLIP algorithm. 4.2 Preprocessing The dataset consists of around 44072 (image,text) pairs. It is divided into three sets of 80% for training, 10% for validation, and 10% for testing. There are 35258 (image,…
Figure 5
Figure 5. Figure 5: represents the t-SNE plot for different fine-tuned model’s image out￾puts in the same latent space obtained by the query ”Give me polo neck t-shirt for men”. Epoch 10, 30, 50, 80, and 100 fine-tuned CLIP models have been selected for this study [PITH_FULL_IMAGE:figure…
Figure 6
Figure 6. Figure 6: K-means Clustering on t-SNE plot for different fine-tuned model’s image out￾puts in same latent space obtained by the query ”Give me polo neck t-shirt for men” [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 18 canonical work pages

  1. [1]

    Aggarwal, P. (2023). Fashion Product Images (Small). Retrieved May 30, 2023, from https://www.kaggle.com/datasets/paramaggarwal/ fashion-product-images-small

    work page 2023

  2. [2]

    L., & Parikh, D

    Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015). VQA: Visual question answering. InIEEE International Conference on Computer Vision (pp. 2425-2433)

    work page 2015

  3. [3]

    K., Hossain, M

    Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010). Multi- modal fusion for multimedia analysis: a survey.Multimedia Systems, 16, 345-379. 14 P.P. Naik and R. Agarwal

    work page 2010

  4. [4]

    Chen, M., & Liu, P. (2017). Performance evaluation of recommender systems.In- ternational Journal of Performability Engineering, 13(8), 1246

    work page 2017

  5. [5]

    J., Attanasio, G., Bianchi, F., Terragni, S., Magalhães, A

    Chia, P. J., Attanasio, G., Bianchi, F., Terragni, S., Magalhães, A. R., Goncalves, D., Greco, C., & Tagliabue, J. (2022). Contrastive language and vision learning of general fashion concepts.Scientific Reports, 12(1), 18958

    work page 2022

  6. [6]

    "Does it come in black?" CLIP-like models are zero-shot recommenders

    Chia, P. J., Tagliabue, J., Bianchi, F., Greco, C., & Goncalves, D. (2022). "Does it come in black?" CLIP-like models are zero-shot recommenders.arXiv preprint arXiv:2204.02473

    work page Pith review arXiv 2022

  7. [7]

    Wikipedia contributors. (2023). Evaluation measures (information retrieval). Re- trieved August 19, 2023, from https://en.wikipedia.org/wiki/Evaluation_ measures_(information_retrieval)

    work page 2023

  8. [8]

    Wikipedia contributors. (2023). Multimodal search. Retrieved August 28, 2023, from https://en.wikipedia.org/wiki/Multimodal_search

    work page 2023

Show all 20 references
  1. [9]

    Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  2. [10]

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., & Dehghani, M. (2020). An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929

    2020 arXiv

  3. [11]

    C., Xu, Y

    Fu, H. C., Xu, Y. Y., & Pao, H. T. (2008). Multimodal search for effective image re- trieval. In15th International Conference on Systems, Signals and Image Processing (pp. 233-236)

    2008

  4. [12]

    Gazzola, P., Pavione, E., Pezzetti, R., & Grechi, D. (2020). Trends in the Fashion Industry. The Perception of Sustainability and Circular Econ- omy: A Gender/Generation Quantitative Approach. Sustainability, 12(7), 2809. https://doi.org/10.3390/su12072809

    2020 doi

  5. [13]

    Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, A., Maschinot, A., Liu, C., & Krishnan, D. (2020). Supervised contrastive learning. InAdvances in Neural Information Processing Systems 33(pp. 18661-18673)

    2020

  6. [14]

    Kofler, C., Larson, M., & Hanjalic, A. (2017). User Intent in Multimedia Search: A Survey of the State of the Art and Future Challenges.ACM Computing Surveys, 49(2), 36. https://doi.org/10.1145/2954930

    2017 doi

  7. [15]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., & Sastry, G

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., & Sastry, G. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning(pp. 8748-8763)

    2021

  8. [16]

    Wang, M., Mahjoubfar, A., & Joshi, A. (2023). FashionVQA: A Domain-Specific Visual Question Answering System. InIEEE/CVF Conference on Computer Vi- sion and Pattern Recognition(pp. 3513-3518)

    2023

  9. [17]

    Sevegnani,K.,Seshadri,A.,Wang,T.,Beniwal,A.,McAuley,J.,Lu,A.,&Medioni, G. (2022). Contrastive learning for interactive recommendation in fashion.arXiv preprint arXiv:2207.12033

    2022 arXiv

  10. [18]

    P., Brocki, Ł., & Marasek, K

    Tautkute, I., Trzciński, T., Skorupa, A. P., Brocki, Ł., & Marasek, K. (2019). Deepstyle: Multimodal search engine for fashion and interior design.IEEE Access, 7, 84613-84628

    2019

  11. [19]

    Sadeh, G., Fritz, L., Shalev, G., & Oks, E. (2019). Joint visual-textual embedding for multimodal style search.arXiv preprint arXiv:1906.06620

    2019 arXiv

  12. [20]

    M., Saputra, D., & Oswari, L

    Saputra, D. M., Saputra, D., & Oswari, L. D. (2020). Effect of distance metrics in determining k-value in k-means clustering using elbow and silhouette method. In Sriwijaya International Conference on Information Technology and Its Applica- tions (SICONIAN 2019)(pp. 341-346)

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.