Pith. sign in

REVIEW 4 major objections 6 minor 23 references

Exploring Visual Embedding Spaces Induced by Vision Transformers for Online Auto Parts Marketplaces

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a Vision Transformer working from images alone can sort online auto-part photos into meaningful, though overlapping, clusters, and that this single-modality capability is a useful but limited foundation for…

desk verdict Clearly written exploratory baseline that honestly reports weak cluster separation, but an internal contradiction about whether the ViT was fine-tuned or frozen leaves the central result attached to an ill-defined model. read the letter →

arxiv 2502.05756 v1 pith:IMIFY33C submitted 2025-02-09 cs.CV cs.LG

Cameron Armijo , Pablo Rivas This is my paper · ORCID
classification cs.CVcs.LG
keywords VisionTransformerautopartsmarketplaceUMAPk-meansclusteringembeddingspacesingle-modalityimageillicitactivitydetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a Vision Transformer, using only images from online auto-parts listings, can produce embeddings whose clusters correspond to meaningful part categories such as full vehicle exteriors, powertrains, body panels, and towing accessories. The authors fine-tune ViT-Base, reduce the 768-dimensional embeddings with UMAP, and run k-means with k=20 on a random sample of 85,000 images scraped from Craigslist and OfferUp. They report that the ten images nearest to each cluster centroid look thematically consistent, which they read as evidence of coherent visual grouping. At the same time, the silhouette score of 0.015 and visible outliers show that the clusters overlap substantially, and the paper's conclusion is that single-modality visual analysis is a useful but limited foundation for detecting stolen or illicit auto parts. The practical motivation is that image-only pipelines could run where text is missing or privacy-sensitive.

What carries the argument

The load-bearing object is the embedding vector $z_i = \mathrm{ViT}(x_i) \in \mathbb{R}^{768}$ produced by the ViT-Base encoder for each image, which is then normalized, reduced to 64 dimensions by UMAP, and partitioned by k-means minimizing the within-cluster variance $J = \sum_{i}\sum_{x \in C_i} \|x-\mu_i\|^2$. The paper's evidence for semantic coherence is the visual consistency of the ten nearest neighbors to each cluster centroid, judged by inspection rather than by labels. UMAP is the mechanism that makes the high-dimensional geometry inspectable, and the clustering indices (silhouette, Calinski-Harabasz, Davies-Bouldin) are the quantitative checks.

What would settle it

Collect a labeled sample of, say, 2,000 images from the same marketplaces with ground-truth part categories, run the paper's exact pipeline (ViT-Base, UMAP to 64 dims, k-means k=20), and compute the adjusted Rand index or cluster purity against those labels; if the agreement is near chance for categories like mirrors vs. body panels, the claimed semantic coherence does not hold.

Watch

Extended reading notes

Core claim

The central claim is that ViT-Base's patch embeddings encode enough visual structure to separate broad families of auto parts without any textual metadata. After UMAP projection to 64 dimensions, k-means with k=20 yields clusters that the authors identify as vehicle exteriors, individual exterior components, powertrains, body panels, and towing accessories, with representative near-centroid images supporting those labels. The authors also find that the same pipeline cannot cleanly separate visually similar parts with different functions, and that mixed-content images create outliers; they quantify this with a silhouette score of 0.015 and a Davies-Bouldin index of 4.164, and they explicitly compare with a multimodal approach that reaches 0.3819.

Load-bearing premise

The interpretation of clusters as auto-part categories rests on assuming that Euclidean distances in the 64-dimensional UMAP map reflect true visual-semantic similarity, with no labeled validation to confirm it.

Editorial extensions

If this is right

  • A vision-only ViT pipeline can separate broad categories of auto parts, so it can serve as a first-pass screener on marketplace images when text is absent or withheld.
  • Cluster overlap and outliers are systematic, not incidental: visually similar but functionally different parts will be mixed, so any downstream illicit-activity detector built on this pipeline needs a text or metadata component.
  • The reported gap in silhouette score (0.015 vs. 0.3819 for a multimodal model) implies that dropping text costs most of the clustering signal in this domain.
  • Fine-tuning on domain-specific auto-part data, outlier filtering, and alternative clustering algorithms are the paper's stated next steps for closing that gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The low silhouette score may reflect UMAP's distortion of global distances rather than a fundamental limit of ViT; sweeping UMAP hyperparameters (n_neighbors, min_dist) and comparing against PCA or t-SNE would isolate where the overlap comes from.
  • The comparison with the multimodal baseline is not controlled: different model, possibly different data, and different reduction choices, so the 0.015 vs. 0.3819 gap should be read as suggestive, not causal.
  • Multi-component listing images could be handled by cropping or object detection before embedding; that would likely remove many outliers and sharpen clusters without any text.
  • The same embedding pipeline, with cluster labels as pseudo-labels, could bootstrap a small labeled set for supervised fine-tuning in domains without existing annotations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies whether Vision Transformer embeddings can group images of auto parts from online marketplaces into semantically meaningful clusters. The authors extract 768-dimensional embeddings from a ViT-Base model for 85,000 images, reduce them with UMAP to 16–128 dimensions, apply k-means with k=20, and report silhouette, Calinski-Harabasz, and Davies-Bouldin scores for each reduced dimension. They then inspect the ten nearest images to selected cluster centroids to argue that clusters correspond to categories such as wheels, lights, bumpers, body panels, and powertrain components. The paper concludes that ViT can produce reasonably coherent clusters but that cluster overlap and outliers reveal limitations of single-modal analysis, and it compares its silhouette score unfavorably with a prior multimodal result.

Significance. If the results were fully supported, the paper would provide a useful empirical data point on the limits of single-modality visual embeddings for a real-world, large-scale marketplace dataset, and its explicit comparison to a multimodal baseline would be relevant to practitioners working on illicit-activity detection. The study has notable strengths: it uses a large, realistic dataset rather than a benchmark; it reports standard clustering metrics across several reduced dimensions; and it candidly acknowledges the low silhouette score and the presence of outliers. However, the central claim that ViT embeddings 'effectively group' auto parts into meaningful categories is not established by the evidence presented. The quantitative metrics indicate almost no cluster separation, and the only qualitative validation is the inspection of a small number of nearest-to-centroid images drawn from the very clusters being evaluated. The paper's contribution is therefore more of an exploratory negative result than a demonstrated capability, and the internal inconsistency about whether the model was fine-tuned weakens the reproducibility of every reported quantity.

major comments (4)
  1. [Overview of ViT-Base Model / Training and Methodology] The manuscript describes the model inconsistently. Section 'Overview of ViT-Base Model / Training' states that 'in this study, ViT is fine-tuned on a car part image classification dataset' and replaces the classification head with y = softmax(Wz_L), while Section 'Methodology / Dataset' says 'no supervised learning involved' and Section 'Methodology / Embeddings' describes a 'pre-trained ViT-Base model' used 'as the feature extractor.' If fine-tuning occurred, the label set, loss function, training data, and validation procedure are never specified, and the dataset section explicitly denies any labeled split. If no fine-tuning occurred, the training section is false. Because the paper's central claim concerns 'visual embedding spaces induced by ViT-Base,' and a fine-tuned ViT induces a different embedding space than a frozen pretrained ViT, this ambiguity affects the interpretation of Table 1, Figures 5–7, and the comparison to the multimodal silhouette score of 0.3819. Please clarify which model variant was actually used and describe its provenance precisely.
  2. [Results and Discussion / Cluster Analysis and Table 1] The silhouette score of 0.0152 for the preferred 64-dimensional configuration is very close to zero, which conventionally indicates that clusters are not separated and that points are nearly as close to other clusters as to their own. Yet Section 'Results and Discussion / Interpretation' claims that ViT's embeddings 'can group visually similar images effectively,' and Section 'Cluster Analysis' asserts that the alignment of nearest-to-centroid images 'supports the validity of our clustering approach.' The paper's own conclusion even says that 'cluster overlap and the presence of outliers revealed key shortcomings.' The claim of 'effective grouping' is therefore contradicted by the quantitative evidence, and the qualitative validation is not independent: it examines images closest to the centroids of the clusters produced by the same algorithm, so near-centroid images will always look mutually similar by construction, even if the overall clustering is poor. To support the central claim, the paper would need either a labeled evaluation set with category agreement measures or a much larger, systematically sampled human evaluation, reported with inter-annotator agreement.
  3. [Methodology / Clustering and Table 1] The selection of k=20 and the reduced dimension of 64 is not adequately justified. The paper says 'the optimal value of k was determined to be 20' but reports no experiment across k values, so the reader cannot assess whether this choice reflects data structure or analyst preference. Similarly, Table 1 varies the reduced dimension but does not report the UMAP hyperparameters (e.g., n_neighbors, min_dist, metric) or the k-means initialization and number of restarts, despite UMAP being a stochastic, hyperparameter-sensitive procedure that can create or destroy cluster structure. Without these details, the reported cluster assignments and the qualitative interpretations in Figures 5–7 are not reproducible, and the comparison across dimensions in Table 1 is not interpretable. Please specify all hyperparameters and seeds, and describe the procedure used to select k.
  4. [Conclusions] The comparison with the multimodal result is not direct. The paper compares its single-modal silhouette score of 0.015 with a multimodal silhouette score of 0.3819 from Hamara and Rivas (2024), but the two studies differ in the embedding model (ViT-Base versus a multimodal model such as ImageBind), the dimensionality-reduction procedure, the clustering algorithm and number of clusters, and quite possibly the sample of images. A cross-study numerical comparison of silhouette scores is not meaningful without controls for these factors. If the comparison is intended to support the paper's claims about the limitations of single-modal models, it should be framed only as suggestive, or ideally the same pipeline should be run on both embedding types with the same preprocessing and clustering choices.
minor comments (6)
  1. [Throughout] The paper alternates between 'single-modality' and 'single-modal'; please choose one term and use it consistently.
  2. [Methodology / Clustering] The paper refers to 'k-Nearest Neighbors (KNN), locating the ten posts nearest to each cluster centroid,' but this is not KNN classification; it is simply a nearest-centroid search. Please rename the procedure (e.g., 'nearest-to-centroid inspection') to avoid confusion.
  3. [Results and Discussion / Figure 4] Figure 4 is a 2D UMAP visualization, but clustering was performed in the 64-dimensional UMAP space. The text should clarify that the 2D plot is for visualization only and may not accurately reflect the distances used by k-means; this is especially important given UMAP's known tendency to distort global geometry.
  4. [Methodology / Dataset] The deduplication step is mentioned but no method is described (e.g., perceptual hashing, embedding similarity, or exact matching). Please specify how duplicates were identified and how many were removed.
  5. [Methodology / Embeddings] The paper does not describe image preprocessing for ViT (resize resolution, normalization, and whether the [CLS] token output or the mean-pooled patch tokens were used as the embedding vector). Since the embedding is the basis of the entire analysis, this information should be stated.
  6. [References] Reference formatting is inconsistent: some arXiv entries include the arXiv identifier and some do not, and some entries lack year or venue details. Please standardize the bibliography.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: the clustering pipeline is externally anchored; only the nearest-to-centroid validation is self-referential.

  1. other [Results and Discussion, 'Cluster Analysis' (validation step near Figures 5-7)]
    "To validate these clusters, we employed k-Nearest Neighbors (KNN), locating the ten posts nearest to each cluster centroid based on Euclidean distance. The alignment of these nearest images with their respective centroids, as displayed in Figures 5, 6, and 7, supports the validity of our clustering approach."

    The ten posts nearest to a k-means centroid are, by construction, the points with the smallest Euclidean distance to that centroid in the same reduced embedding space used to define the clusters. Their proximity to the centroid is therefore guaranteed by the clustering algorithm, not an independent confirmation that the cluster boundaries are semantically meaningful. Showing these images as 'support' verifies only that the core of each cluster is self-consistent in the embedding space that produced the clusters, making the qualitative validation self-referential rather than an external check.

full rationale

The paper's central derivation is not circular: it takes embeddings from an external pretrained ViT encoder (Sec. 'Embeddings'), reduces them with UMAP, applies k-means, and reports internal clustering metrics. None of these quantities is an input that forces the stated conclusion; the silhouette, Calinski-Harabasz, and Davies-Bouldin indices are conventional internal evaluations on the same data, which is standard practice. The paper makes no predictive claim that is fitted to a subset and then 'predicted' on a closely related quantity. The only self-referential element is the cluster validation: the ten nearest images to each k-means centroid are, by construction, the points closest to that centroid in the same reduced space used to define the clusters, so their proximity cannot independently confirm semantic coherence. This is a mild circularity in the validation narrative, not in the quantitative derivation. The comparison to the 0.3819 multimodal silhouette is imported from the authors' prior work (Hamara and Rivas 2024); it is a self-citation used as a benchmark, but it does not feed back into the present cluster computation, so it does not make the central result circular. Also noted but not circular: the paper is internally inconsistent about whether the ViT was fine-tuned or used frozen, which is a model-identity and correctness risk, not a circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the transferability of ImageNet-pretrained ViT features to auto parts images, on UMAP reduction preserving the structure needed for k-means, and on subjective visual validation of clusters. The free parameters are the cluster count and UMAP settings, both chosen empirically without stability analysis.

free parameters (3)
  • number of clusters k = 20
    Chosen after experimentation as 'optimal' balancing cohesion and separation, no selection procedure or stability analysis given.
  • UMAP reduced dimension = 64
    Selected as best trade-off from {16,32,64,128} based on silhouette, Calinski-Harabasz, and Davies-Bouldin; difference from 128 is small.
  • UMAP hyperparameters = unspecified
    n_neighbors, min_dist, and metric are not reported, so the embedding is not uniquely determined.
assumptions (4)
  • domain assumption Pretrained ViT-Base embeddings generalize to auto parts images
    The paper uses the pretrained model's output as a feature extractor without any fine-tuning (Methodology, Embeddings), assuming ImageNet-trained features transfer to this domain.
  • domain assumption UMAP reduced embeddings preserve cluster structure
    k-means is applied after UMAP reduction; UMAP is known to distort global distances and can introduce or hide cluster structure.
  • standard math K-means spherical cluster assumption
    K-means assumes convex, equally sized clusters; the paper acknowledges this in Methodology, Clustering.
  • ad hoc to paper Nearest-to-centroid images are representative of cluster content
    Cluster validation relies on visually inspecting ten nearest images per centroid without ground truth labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Visual Embedding Spaces Induced by Vision Transformers for Online Auto Parts Marketplaces." pith.science (2026). https://pith.science/paper/IMIFY33C

@misc{pith2026250205756,
  author       = {Pith},
  title        = {Pith review of: Exploring Visual Embedding Spaces Induced by Vision Transformers for Online Auto Parts Marketplaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IMIFY33C}},
  note         = {Machine review of arXiv:2502.05756}
}
read the original abstract

This study examines the capabilities of the Vision Transformer (ViT) model in generating visual embeddings for images of auto parts sourced from online marketplaces, such as Craigslist and OfferUp. By focusing exclusively on single-modality data, the analysis evaluates ViT's potential for detecting patterns indicative of illicit activities. The workflow involves extracting high-dimensional embeddings from images, applying dimensionality reduction techniques like Uniform Manifold Approximation and Projection (UMAP) to visualize the embedding space, and using K-Means clustering to categorize similar items. Representative posts nearest to each cluster centroid provide insights into the composition and characteristics of the clusters. While the results highlight the strengths of ViT in isolating visual patterns, challenges such as overlapping clusters and outliers underscore the limitations of single-modal approaches in this domain. This work contributes to understanding the role of Vision Transformers in analyzing online marketplaces and offers a foundation for future advancements in detecting fraudulent or illegal activities.

Figures

Figures reproduced from arXiv: 2502.05756 by the authors.

Figure 1
Figure 1. Example of an interior view from an online auto [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Vision Transformer architecture, illustrating the process of dividing an input image into patches, applying a linear [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. In the proposed methodology, the input data are images that are embedded with a ViT and then analyzed in search of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: UMAP visualization of embeddings reduced to 64 dimensions, illustrating the clustering of images from online auto [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Representative images from posts located near a [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 5
Figure 5. Figure 5: Representative images from posts located near a [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 8
Figure 8. Figure 8: Examples of images from cluster 3, showcasing [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 11 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alayrac, J.-B.; et al. 2023. Flamingo: a Visual Language Model for Few-Shot Learning. In Advances in Neural Information Processing Systems (NeurIPS)

    work page 2023

  4. [4]

    Caliński, T.; and Harabasz, J. 1974. A dendrite method for cluster analysis. Communications in Statistics, 3(1): 1--27

    work page 1974

  5. [5]

    Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020. End-to-End Object Detection with Transformers. arXiv:2005.12872

    arXiv 2020

  6. [6]

    Cordonnier, J.-B.; Loukas, A.; and Jaggi, M. 2020. On the Relationship between Self-Attention and Convolutional Layers. arXiv:1911.03584

    arXiv 2020

  7. [7]

    L.; and Bouldin, D

    Davies, D. L.; and Bouldin, D. W. 1979. A Cluster Separation Measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-1(2): 224--227

    work page 1979

  8. [8]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929

    arXiv 2021

Show all 23 references
  1. [9]

    Ester, M.; Kriegel, H.-P.; Sander, J.; Xu, X.; et al. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, 226--231

    1996

  2. [10]

    V.; Joulin, A.; and Misra, I

    Girdhar, R.; El-Nouby, A.; Liu, Z.; Singh, M.; Alwala, K. V.; Joulin, A.; and Misra, I. 2023. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15180--15190

    2023

  3. [11]

    J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y

    Goodfellow, I. J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative Adversarial Networks. arXiv:1406.2661

    2014 arXiv

  4. [12]

    Hamara, A.; and Rivas, P. 2024. From Latent to Engine Manifolds: Analyzing ImageBind's Multimodal Embedding Space. arXiv:2409.10528

    2024 arXiv

  5. [13]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 770--778

    2016

  6. [14]

    McInnes, L.; Healy, J.; and Melville, J. 2020. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv:1802.03426

    2020 arXiv

  7. [15]

    Murtagh, F.; and Contreras, P. 2012. Algorithms for hierarchical clustering: an overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2(1): 86--97

    2012

  8. [16]

    Radford, A.; Metz, L.; and Chintala, S. 2016. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv:1511.06434

    2016 arXiv

  9. [17]

    Radford, A.; and Narasimhan, K. 2018. Improving Language Understanding by Generative Pre-Training

    2018

  10. [18]

    B.; and Rivas, P

    Rashid, M. B.; and Rivas, P. 2024. AI Safety in Practice: Enhancing Adversarial Robustness in Multimodal Image Captioning. arXiv:2407.21174

    2024 arXiv

  11. [19]

    Rousseeuw, P. J. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20: 53--65

    1987

  12. [20]

    A.; Stricker, D.; and Afzal, M

    Shehzadi, T.; Hashmi, K. A.; Stricker, D.; and Afzal, M. Z. 2023. Object Detection with Transformers: A Review. arXiv:2306.04670

    2023 arXiv

  13. [21]

    Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and J \'e gou, H. 2021. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML), 10347--10357

    2021

  14. [22]

    N.; Kaiser, L.; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2023. Attention Is All You Need. arXiv:1706.03762

    2023 arXiv

  15. [23]

    Wu, B.; Xu, C.; Dai, X.; Wan, A.; Zhang, P.; Yan, Z.; Tomizuka, M.; Gonzalez, J.; Keutzer, K.; and Vajda, P. 2020. Visual Transformers: Token-based Image Representation and Processing for Computer Vision. arXiv:2006.03677

    2020 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.