Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

In a benchmark of 75 queries over 3,612 digitized textbook pages, cosine similarity retrieved the most semantically and visually relevant pages.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 21:00 UTC pith:5EHZED6O

load-bearing objection A modest, honestly-reported benchmark study whose central cosine-vs-dot-product claim is under-supported by the evidence it presents. the 4 major comments →

arxiv 2509.08216 v1 pith:5EHZED6O submitted 2025-09-10 cs.IR

Vector embedding of multi-modal texts: a tool for discovery?

Beth Plale , Sai Navya Jyesta , Sachith Withana This is my paper
classification cs.IR
keywords vector embeddingmulti-modal document retrievaldigital library discoverycosine similarityvision-language modellate interaction scoringColPalivector database benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether vector embeddings of page images—not just their OCR text—can help people discover relevant content in digitized book collections. Using a vision-language model (ColPali) to represent each page as a multi-vector embedding, the authors issue 75 natural-language queries and compare four distance measures against a manually curated ground truth. They report that cosine similarity retrieves the most relevant pages, with the highest scores on precision, recall, F1, average precision, and mean reciprocal rank. They also find that the system surfaces relevant pages the ground truth missed, which suggests label-based evaluation understates the discovery potential of multimodal vector retrieval.

Core claim

The central empirical claim is that cosine similarity is the most effective distance measure for ColPali-style multi-vector page embeddings in a digital-library setting. With the embedding model, query set, and index held constant, only the similarity function changes across four Qdrant collections; under that controlled comparison, cosine achieves the highest Precision@5 (0.514), Recall@5 (0.281), F1@5 (0.353), Average Precision (0.238), and MRR (0.801) on the full 75-query benchmark, and it also leads on the multi-page and conceptual query subsets. The authors further report that for 18 of 75 queries the system retrieves high-scoring pages outside the gold-standard set, with 72% of those p

What carries the argument

ColPali, a vision-language model that divides each page image into 1,030 visual patches and projects them into a shared 128-dimensional space, producing a multi-vector (1030×128) representation per page. The retrieval mechanism is late-interaction scoring (introduced by ColBERT): each query token scores against every page patch, and max-sum pooling aggregates these scores into a page relevance value. This multi-vector, late-interaction setup is what lets the authors isolate the effect of the four distance functions, since the embeddings and query set are identical across the four Qdrant collections.

Load-bearing premise

The comparison of distance measures rests on the manually curated ground truth for the 75 queries being both complete and unbiased; if many relevant pages are missing or mislabeled, every reported metric difference is open to question.

What would settle it

Re-run the 75-query benchmark with an expanded ground truth that includes the 18 relevant-but-unlabeled pages found in Experiment IV plus any additional pages a second, independent annotator marks as relevant. If cosine similarity's Precision@5 no longer beats dot product's by a meaningful margin, the claim that cosine is the most effective metric is not robust to ground-truth completeness.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Adopt cosine similarity as the default distance metric for ColPali/Qdrant page-image retrieval in visually dense collections; it beat the other three metrics on every benchmark.
  • A single natural-language query can satisfy multi-modal information needs: the top-five results often include both a diagram and its explanatory text.
  • Top-k evaluation against a manually curated gold standard understates the system's usefulness, since 18 of 75 queries produced relevant pages the annotators had not labeled.
  • Late interaction means page embeddings are computed once and stored, so the operational bottleneck in a large collection shifts to query-embedding concurrency—flagged in the paper as future work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because cosine's Precision@5 is 0.514, roughly half of the top five results are not relevant; the limiting component may be the max-sum aggregation rather than the embedding model, so a learned re-ranking step could be the next improvement to test.
  • The unlabeled-but-relevant hits (18 of 75 queries) suggest a concrete discovery feature: a 'surprise' view that shows users high-similarity pages outside the known relevant set, since 72% of such pages were validated as relevant.
  • The ground-truth incompleteness cuts both ways: expanding the gold set with the missed relevant pages could widen, shrink, or even reverse the cosine-vs-dot-product margin (0.514 vs 0.496 in Precision@5), so the authors' planned ground-truth strengthening is the decisive test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes a multimodal retrieval pipeline in which page images from about 3,600 digitized computer-science textbook pages are embedded with the ColPali vision-language model into multi-vector representations and stored in the Qdrant vector database. A benchmark of 75 natural-language queries is used to compare four similarity functions (cosine, dot product, Euclidean, Manhattan) under an otherwise fixed pipeline, using Precision@5, Recall@5, F1@5, AP, and MRR. Three experiments compare metrics on the full benchmark and on multi-page/conceptual subsets; a fourth 'unbounded' experiment looks for relevant pages outside the manually curated ground truth. The central claim is that cosine similarity most effectively retrieves semantically and visually relevant pages, and that multimodal vector retrieval can augment discovery in digital libraries.

Significance. If the comparative claim were statistically supported, the paper would offer a useful, practical design insight for digital-library discovery over visually rich documents. The study's strengths include a controlled experimental design that varies only the distance function; release of the benchmark and code (Zenodo/GitHub links in Sec. VII); and a clearly described use of ColPali and late-interaction scoring. The serendipity experiment (Experiment IV) is a genuinely interesting attempt to move beyond fixed labels. However, the load-bearing statistical evidence is currently missing, and the acknowledged incompleteness of the ground truth directly affects the central comparison. The manuscript is a plausible basis for a workshop or short paper, but as a journal contribution it needs additional analysis to substantiate the abstract's claim.

major comments (4)
  1. [§IV.D, Table II; §V] The Discussion opens 'We have shown statistically that the strongest distance measure, cosine similarity...' but no significance test, confidence interval, or variance estimate appears anywhere in the paper. In Table II the cosine-vs-dot-product gaps are Prec@5 0.514 vs 0.496, Recall@5 0.281 vs 0.272, and MRR 0.801 vs 0.787. With 75 queries and no error bars, these differences are within plausible sampling variation. A paired bootstrap or Wilcoxon signed-rank test over the 75 queries, together with effect sizes, is needed before the abstract's comparative ranking can be accepted.
  2. [§IV.B, §IV.D.d, §V] The paper concedes in §V that the ground truth is incomplete ('The next step in this work is strengthen the completeness and relevance of the ground truth itself'). Experiment IV then reports that 18 of 75 queries return relevant-but-unlabeled pages, with 72% validated by human annotators and GPT-4, but it does not report how these pages are distributed across cosine and dot product, nor the basis for the >0.85 similarity threshold. Because the observed metric gaps are small, adding even a few dot-product-only relevant pages to the label set could change the ranking. Please report per-metric counts and re-run Tables II–IV with the expanded labels; also report inter-annotator agreement for the ground-truth construction and for the Experiment IV validation.
  3. [§I, §V, §VI] The motivating claim is that vector retrieval can 'augment a digital library discovery strategy' beyond 'retrieval techniques currently in place' (§I). Yet no comparison to BM25, full-text search, or a text-only embedding baseline is provided. The current experiments only compare distance functions within a single ColPali/Qdrant pipeline. Without a text-only or keyword baseline, the broader discovery-augmentation claim is not tested. Adding such a baseline would also contextualize the absolute Precision@5 values (0.514 for cosine), which are modest.
  4. [§IV.D.d] Experiment IV contains an internal inconsistency. It first says, 'We illustrate results in Figure 2 using a single benchmark query. The result is from query NU-2 (numerical query subset, query #2): What is the cost difference between the path found by the weighted A* search and the path found by standard A*...' and then immediately says, 'The query, presented first in Sec II, is this: "Which page explains the intuition behind eventual consistency?".' These are different queries; the latter is not in Sec. II and is not numerical. This makes it unclear which query produced Figure 2 and which query the surrounding discussion of distributed clocks and causal ordering refers to. Please correct the text and ensure the figure caption matches the query actually used.
minor comments (5)
  1. [Abstract] Typo: 'weakenesses' should be 'weaknesses'.
  2. [§III.A.a] 'PaliGemma-3Ba' should likely be 'PaliGemma-3B'.
  3. [§III.A.c] Formatting artifact: 'T op−5' should be 'Top-5'.
  4. [References] References [10] and [11] are the same ColBERT paper; please deduplicate. Reference [3] title should be capitalized consistently ('PaliGemma: A Versatile 3B VLM for Transfer').
  5. [§II] The use-case queries are labeled MP-14, MM-1, and CP-13, but the code used for these labels is not defined in the text and does not match the benchmark category names in Table I. Please explain the labeling or remove it.

Circularity Check

0 steps flagged

No significant circularity: the central cosine-similarity claim is an empirical benchmark comparison against a manually curated ground truth, not a fitted input or self-referential derivation.

full rationale

The paper's central claim—that cosine similarity most effectively retrieves semantically and visually relevant pages—is an empirical finding from a controlled comparison of four distance functions (Cosine, Dot Product, Euclidean, Manhattan) under a fixed embedding model (ColPali/PaliGemma-3B), fixed corpus, fixed query set, and fixed evaluation protocol. The ground truth is described as a manually curated set of query–passage–relevance triples (Section IV.B), constructed independently of the similarity functions being compared. No parameter is fitted from the ground truth and then used to predict the same quantity; the comparison is made against the same fixed labels for all metrics. The paper explicitly acknowledges the ground truth is incomplete (Section V: 'The next step in this work is strengthen the completeness and relevance of the ground truth itself'), which is a validity limitation, not a circularity. Experiment IV examines relevant-but-unlabeled pages, but these are validated post hoc by human annotators and GPT-4 and are not fed back into the metric comparison; it is an exploratory analysis, not a fitted predictor. Self-citations (the benchmark data repository [12] and code repository [13]) are artifacts of the study, not load-bearing evidence for the cosine claim. The absence of significance tests for the small cosine-vs-dot-product margins is a statistical rigor concern, not a circularity. No equation or definition reduces the claimed finding to its own inputs, and no load-bearing argument rests on a self-citation or an imported uniqueness theorem. Hence no circularity is present.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The paper introduces no new theoretical entities. Its central results rest on empirical configuration choices: the manually built ground truth, the pretrained ColPali embedding model, the 0.85 similarity threshold used only in the serendipity experiment, and the assumed representativeness of five textbooks. These are domain assumptions rather than fitted parameters in a derivation, but they are load-bearing for the reported metrics.

free parameters (1)
  • Similarity threshold for 'high-scoring' pages in Experiment IV = 0.85
    Pages with Cosine or Dot Product similarity greater than 0.85 are called relevant-but-unlabeled; the threshold is stated without justification and directly affects the 72% validation figure.
axioms (4)
  • domain assumption Manually curated ground truth labels accurately reflect relevance for each query.
    All metrics are computed against this ground truth; the paper admits it is assembled by graduate students and needs strengthening (Section V).
  • domain assumption ColPali/PaliGemma-3B embeddings capture both textual and visual semantics sufficiently for retrieval.
    The entire approach depends on VLM embeddings representing page content; there is no independent validation of embedding quality beyond retrieval performance.
  • domain assumption The corpus of 3,612 textbook pages is representative of digital library content.
    Generalizations to collections like HathiTrust assume this small textbook corpus is representative (Sections I and V).
  • domain assumption Max-sum pooling across late interaction scores is an appropriate aggregation.
    Adopted from ColBERT and ColPali without ablation; retrieval scores depend on this pooling choice (Section III.A).

pith-pipeline@v1.3.0-alltime-deepseek · 11104 in / 11491 out tokens · 113620 ms · 2026-08-04T21:00:19.168717+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Vector embedding of multi-modal texts: a tool for discovery?." pith.science (2026). https://pith.science/paper/5EHZED6O

@misc{pith2026250908216,
  author       = {Pith},
  title        = {Pith review of: Vector embedding of multi-modal texts: a tool for discovery?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5EHZED6O}},
  note         = {Machine review of arXiv:2509.08216}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Computer science texts are particularly rich in both narrative content and illustrative charts, algorithms, images, annotated diagrams, etc. This study explores the extent to which vector-based multimodal retrieval, powered by vision-language models (VLMs), can improve discovery across multi-modal (text and images) content. Using over 3,600 digitized textbook pages largely from computer science textbooks and a Vision Language Model (VLM), we generate multi-vector representations capturing both textual and visual semantics. These embeddings are stored in a vector database. We issue a benchmark of 75 natural language queries and compare retrieval performance to ground truth and across four similarity (distance) measures. The study is intended to expose both the strengths and weakenesses of such an approach. We find that cosine similarity most effectively retrieves semantically and visually relevant pages. We further discuss the practicality of using a vector database and multi-modal embedding for operational information retrieval. Our paper is intended to offer design insights for discovery over digital libraries. Keywords: Vector embedding, multi-modal document retrieval, vector database benchmark, digital library discovery

Figures

Figures reproduced from arXiv: 2509.08216 by Beth Plale, Sachith Withana, Sai Navya Jyesta.

Figure 1
Figure 1. Figure 1: Architectural Schematic The design is guided by the need for rigorous experimental control, where all aspects of the retrieval pipeline are held constant across all experiments, including embedding model, dataset, indexing method, query formulation, and evaluation protocol. The only variable that can be altered between runs is the choice of similarity function used during search of the vector database. Thi… view at source ↗
Figure 2
Figure 2. Figure 2: Page distance for numerical query NU-2 We conduct this evaluation using both Cosine similarity and Dot Product which are the top two performing distance functions from our earlier experiments. For each query, we identify pages returned outside the Top-5 that exhibit strong semantic alignment with the query but are not present in the original ground truth set. To assess their value, we: • 1) Measure similar… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Qiskit Code Migration with LLMs

    cs.SE 2026-06 unverdicted novelty 5.0

    A taxonomy-guided RAG system with LLMs reduces hallucinations and improves migration suggestions for Qiskit code compared to unconstrained retrieval.

Reference graph

Works this paper leans on

17 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Getting ViT in shape: Scaling laws for compute- optimal model design

    Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting ViT in shape: Scaling laws for compute- optimal model design. arXiv preprint arXiv:2305.13035, 2023. https://arxiv.org/abs/2305.13035

    Pith/arXiv arXiv 2023

  2. [2]

    Manmatha

    Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. DocFormer: End-to-end transformer for document understanding. arXiv preprint arXiv:2106.11539, 2021. https://arxiv.org/abs/2106.11539

    Pith/arXiv arXiv 2021

  3. [3]

    Paligemma: A versatile 3b vlm for transfer

    Lucas Beyer et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024. https://arxiv.org/abs/2407.07726

    Pith/arXiv arXiv 2024

  4. [4]

    Imagined visual representations as multimodal embeddings.Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb

    Guillem Collell, Ted Zhang, and Marie-Francine Moens. Imagined visual representations as multimodal embeddings.Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017

    work page 2017

  5. [5]

    Introduction to and hands-on use cases with hathitrust research center’s extracted features 2.0 dataset

    Ryan Dubnicek and Deren Kudeki. Introduction to and hands-on use cases with hathitrust research center’s extracted features 2.0 dataset. In 2021 ACM/IEEE Joint Conf on Digital Libraries (JCDL), pages 352– 353, 2021

    work page 2021

  6. [6]

    ColPali: Efficient Document Retrieval with Vision Language Models.arXiv preprint arXiv:2407.01449, 2024

    Manuel Faysse et al. ColPali: Efficient Document Retrieval with Vision Language Models.arXiv preprint arXiv:2407.01449, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Colpali: Efficient document retrieval with vision language models

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Celine Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. InThe 13th Intl Conf on Learning Representations. Open Review.net, 2025

    work page 2025

  8. [8]

    Layoutlmv3: Pre-training for document ai with unified text and image masking

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. arXiv preprint arXiv:2204.08387, 2022. https://arxiv.org/abs/2204.08387

    Pith/arXiv arXiv 2022

  9. [9]

    Copyright violations and large language models

    Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models. InProc 2023 Conf on Empirical Methods in Natural Language Processing, page 7403–7412. Association for Computational Linguistics, 2023

    work page 2023

  10. [11]

    Colbert: Efficient and effective passage search via contextualized late interaction over bert

    Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. arXiv preprint arXiv:2004.12832, 2020. https://arxiv.org/abs/2004.12832

    Pith/arXiv arXiv 2004

  11. [12]

    Plale and S

    B. Plale and S. Withana. Vector discovery benchmark 75 result [data set]. Zenodo, 2025. https://doi.org/10.5281/zenodo.17064743

    work page doi:10.5281/zenodo.17064743 2025

  12. [13]

    Plale, S

    B. Plale, S. Withana, and Sia Navya Jyesta. Vector discovery bench- mark code repo. GitHub, 2025. https://github.com/Data-to-Insight- Center/vector-volume-discovery-pipeline

    work page 2025

  13. [14]

    Vector database, 2021

    Qdrant. Vector database, 2021. https://qdrant.tech/

    work page 2021

  14. [15]

    Robertson, Steve Walker, Susan Jones, Micheline Hancock- Beaulieu, and Mike Gatford

    Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock- Beaulieu, and Mike Gatford. Okapi at TREC-3. InProceedings of The Third Text REtrieval Conference, TREC 1994, volume 500-225 ofNIST Special Publication, pages 109–126, Gaithersburg, Maryland, USA, November 1994. National Institute of Standards and Technology (NIST). November 2-4, 1994

    work page 1994

  15. [16]

    A Statistical Interpretation of Term Specificity and its Application in Retrieval.Journal of Documentation, 28(1):11–21, 1972

    Karen Sparck Jones. A Statistical Interpretation of Term Specificity and its Application in Retrieval.Journal of Documentation, 28(1):11–21, 1972

    work page 1972

  16. [17]

    Gemma: Open Models Based on Gemini Research and Technology

    Gemma Team et al. Gemma: Open Models Based on Gemini Research and Technology. arXiv preprint arXiv:2403.08295, 2024. https://arxiv.org/abs/2403.08295

    Pith/arXiv arXiv 2024

  17. [18]

    Wilkinson, M

    M. Wilkinson, M. Dumontier, and I. et al. Aalbersberg. The fair guiding principles for scientific data management and stewardship

    work page