{"id":"d3666450-29e6-4990-bb28-dc0e9995a1cc","arxiv_id":"2509.08216","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Using ColPali embeddings of 3,600 textbook page images, cosine similarity beats dot product, Euclidean, and Manhattan distances on top-5 retrieval, but only reaches 0.51 precision@5 without a text-only baseline.","lead":"This paper tests whether AI-powered vector search over page images, including both text and diagrams, helps find relevant pages in digitized computer science textbooks. Across 75 queries and four distance measures, cosine similarity performed best, offering practical design guidance for digital library discovery.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth incompleteness could differentially shift the cosine vs dot product margin; the paper's claim of statistical support is unsupported by any significance test.","rationale":"The paper's central claim is a comparative one: cosine is the most effective similarity measure among the four tested. For that claim to hold, two conditions must be met: (1) the evaluation labels must be reliable enough that observed differences reflect true retrieval quality, and (2) the observed differences must be larger than what random variation would produce. The second condition is not addressed at all; the first is explicitly acknowledged as imperfect. The small margins in Table II (≤0.018 in all metrics except MRR where the margin is 0.014) mean that even modest label noise could change the ranking. Experiment IV shows that the ground truth has false negatives—relevant pages that are unlabeled. If those false negatives are not uniformly distributed across the four collections (i.e., if dot product tends to rank unlabeled relevant pages higher than cosine does), then the measured advantage of cosine is an artifact of incomplete labeling. The paper does not report the breakdown, so the concern is live. The proposed test directly addresses this by turning the unlabeled-but-validated pages into labels and re-running the comparison. The significance test is included because the paper's verbal claim of statistical support is not backed by any statistic; without it, even the current small margin cannot be distinguished from noise. This is the single most load-bearing concern because it strikes at the comparative claim itself, and it is testable with data already in hand. I agree with the reader's identification of the ground-truth issue; my concern refines it by showing why it could specifically affect the cosine-vs-dot comparison, rather than merely lowering absolute scores.","tokens_in":11353,"tokens_out":6357,"duration_ms":72247,"concrete_test":"Recompute Table II after augmenting the ground truth with the 18 relevant-but-unlabeled pages identified in Experiment IV (and any others found by the annotators), allocating them to the queries and pages where they were validated. Then perform a paired bootstrap or Wilcoxon signed-rank test on per-query AP and Prec@5 comparing cosine vs dot product. If the cosine advantage is not significantly different from zero, or if dot product's recall gains exceed cosine's such that it ties or surpasses cosine, the abstract claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'cosine similarity most effectively retrieves semantically and visually relevant pages' rests on the metric comparisons in Table II, which depend entirely on the manually curated ground truth. The paper itself concedes this ground truth is incomplete (Section V: 'The next step in this work is strengthen the completeness and relevance of the ground truth itself'). Experiment IV then demonstrates that 18 of 75 queries return relevant-but-unlabeled pages, validated by human annotators and GPT-4, but the paper does not report how these unlabeled relevant pages are distributed across the distance measures. Because the observed advantages of cosine over dot product are small (Prec@5: 0.514 vs 0.496; Recall@5: 0.281 vs 0.272; MRR: 0.801 vs 0.787), the inclusion of even a handful of dot-product-retrieved unlabeled relevant pages could erode or reverse the gap. Moreover, the Discussion asserts 'We have shown statistically' that cosine is strongest, but no significance test, confidence interval, or variance estimate appears anywhere in the paper. Thus the abstract's comparative claim is not established even for the current ground truth. The load-bearing assumption is that the curated labels are both complete enough and unbiased enough to support the small observed differences; this is the least secure condition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a multimodal retrieval pipeline in which page images from about 3,600 digitized computer-science textbook pages are embedded with the ColPali vision-language model into multi-vector representations and stored in the Qdrant vector database. A benchmark of 75 natural-language queries is used to compare four similarity functions (cosine, dot product, Euclidean, Manhattan) under an otherwise fixed pipeline, using Precision@5, Recall@5, F1@5, AP, and MRR. Three experiments compare metrics on the full benchmark and on multi-page/conceptual subsets; a fourth 'unbounded' experiment looks for relevant pages outside the manually curated ground truth. The central claim is that cosine similarity most effectively retrieves semantically and visually relevant pages, and that multimodal vector retrieval can augment discovery in digital libraries.","tokens_in":11669,"tokens_out":3586,"duration_ms":43346,"significance":"If the comparative claim were statistically supported, the paper would offer a useful, practical design insight for digital-library discovery over visually rich documents. The study's strengths include a controlled experimental design that varies only the distance function; release of the benchmark and code (Zenodo/GitHub links in Sec. VII); and a clearly described use of ColPali and late-interaction scoring. The serendipity experiment (Experiment IV) is a genuinely interesting attempt to move beyond fixed labels. However, the load-bearing statistical evidence is currently missing, and the acknowledged incompleteness of the ground truth directly affects the central comparison. The manuscript is a plausible basis for a workshop or short paper, but as a journal contribution it needs additional analysis to substantiate the abstract's claim.","major_comments":[{"comment":"The Discussion opens 'We have shown statistically that the strongest distance measure, cosine similarity...' but no significance test, confidence interval, or variance estimate appears anywhere in the paper. In Table II the cosine-vs-dot-product gaps are Prec@5 0.514 vs 0.496, Recall@5 0.281 vs 0.272, and MRR 0.801 vs 0.787. With 75 queries and no error bars, these differences are within plausible sampling variation. A paired bootstrap or Wilcoxon signed-rank test over the 75 queries, together with effect sizes, is needed before the abstract's comparative ranking can be accepted.","section":"§IV.D, Table II; §V"},{"comment":"The paper concedes in §V that the ground truth is incomplete ('The next step in this work is strengthen the completeness and relevance of the ground truth itself'). Experiment IV then reports that 18 of 75 queries return relevant-but-unlabeled pages, with 72% validated by human annotators and GPT-4, but it does not report how these pages are distributed across cosine and dot product, nor the basis for the >0.85 similarity threshold. Because the observed metric gaps are small, adding even a few dot-product-only relevant pages to the label set could change the ranking. Please report per-metric counts and re-run Tables II–IV with the expanded labels; also report inter-annotator agreement for the ground-truth construction and for the Experiment IV validation.","section":"§IV.B, §IV.D.d, §V"},{"comment":"The motivating claim is that vector retrieval can 'augment a digital library discovery strategy' beyond 'retrieval techniques currently in place' (§I). Yet no comparison to BM25, full-text search, or a text-only embedding baseline is provided. The current experiments only compare distance functions within a single ColPali/Qdrant pipeline. Without a text-only or keyword baseline, the broader discovery-augmentation claim is not tested. Adding such a baseline would also contextualize the absolute Precision@5 values (0.514 for cosine), which are modest.","section":"§I, §V, §VI"},{"comment":"Experiment IV contains an internal inconsistency. It first says, 'We illustrate results in Figure 2 using a single benchmark query. The result is from query NU-2 (numerical query subset, query #2): What is the cost difference between the path found by the weighted A* search and the path found by standard A*...' and then immediately says, 'The query, presented first in Sec II, is this: \"Which page explains the intuition behind eventual consistency?\".' These are different queries; the latter is not in Sec. II and is not numerical. This makes it unclear which query produced Figure 2 and which query the surrounding discussion of distributed clocks and causal ordering refers to. Please correct the text and ensure the figure caption matches the query actually used.","section":"§IV.D.d"}],"minor_comments":[{"comment":"Typo: 'weakenesses' should be 'weaknesses'.","section":"Abstract"},{"comment":"'PaliGemma-3Ba' should likely be 'PaliGemma-3B'.","section":"§III.A.a"},{"comment":"Formatting artifact: 'T op−5' should be 'Top-5'.","section":"§III.A.c"},{"comment":"References [10] and [11] are the same ColBERT paper; please deduplicate. Reference [3] title should be capitalized consistently ('PaliGemma: A Versatile 3B VLM for Transfer').","section":"References"},{"comment":"The use-case queries are labeled MP-14, MM-1, and CP-13, but the code used for these labels is not defined in the text and does not match the benchmark category names in Table I. Please explain the labeling or remove it.","section":"§II"}],"recommendation":"major_revision","confidential_remarks":"The paper releases its benchmark and code, which is commendable and makes the requested additional analyses feasible. The main risk is not the novelty of the pipeline but the gap between the reported small metric differences and the strong comparative claim. I see no evidence of misconduct; the issues are analyzable and fixable within the manuscript's scope. The paper may be suitable for the journal after the authors add significance testing, a text-only baseline, and a per-metric breakdown of the Experiment IV unlabeled relevant pages."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a straightforward application of ColPali to a 3,600-page textbook corpus, with a 75-query benchmark and manually curated ground truth. The genuinely useful pieces are the released benchmark and code, and the honest acknowledgment that gold-standard labels miss relevant pages.\n\nWhat's new: the specific benchmark, the comparison of four distance measures on this multimodal setup, and the 'serendipity' experiment that surfaces unlabeled relevant pages. That's a modest but real contribution for people building discovery systems over digitized books.\n\nThe experimental design is clean in one respect: they isolate the distance metric, holding embeddings, index, and queries constant. The paper is also transparent about the ground truth being incomplete, which is more than many papers do.\n\nThe central claim, however, is not actually established. The margins over dot product are small (Precision@5 0.514 vs 0.496, MRR 0.801 vs 0.787), and there is no significance test, confidence interval, or variance estimate anywhere. The Discussion's 'We have shown statistically' is unjustified. Second, the evaluation lacks a text-only retrieval baseline (BM25 or full-text search), so the opening framing that multimodal retrieval improves discovery is untested. Third, the ground truth is self-curated, and the paper concedes it needs strengthening. The stress-test note is right that the cosine margin could shift with a more complete relevance set, though I'd add that the direction is not obvious; it could go either way. The bigger issue is that the claim is oversold relative to the evidence.\n\nAlso, there is a mislabel in Experiment IV: they describe query NU-2 as the A* cost-difference question, then immediately illustrate it with the 'eventual consistency' query without a clear segue. That kind of copy-paste error suggests a rushed final pass.\n\nWho is this for? Technologists in digital libraries who want a quick read on whether ColPali/Qdrant is worth trying; they'll get a reasonable proof-of-concept and a benchmark to reuse. IR researchers will find the evaluation too thin to trust the headline result.\n\nWorth a serious referee? I'd say yes, but conditional on the authors adding significance testing or at least error bars, including a BM25 baseline, and fixing the Experiment IV mislabel. The benchmark and code availability make it worth engaging with.","headline":"A modest, honestly-reported benchmark study whose central cosine-vs-dot-product claim is under-supported by the evidence it presents.","tokens_in":12101,"tokens_out":1843,"would_cite":true,"duration_ms":21810,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a benchmark of 75 queries over 3,612 digitized textbook pages, cosine similarity retrieved the most semantically and visually relevant pages.","keywords":["vector embedding","multi-modal document retrieval","digital library discovery","cosine similarity","vision-language model","late interaction scoring","ColPali","vector database benchmark"],"falsifier":"Re-run the 75-query benchmark with an expanded ground truth that includes the 18 relevant-but-unlabeled pages found in Experiment IV plus any additional pages a second, independent annotator marks as relevant. If cosine similarity's Precision@5 no longer beats dot product's by a meaningful margin, the claim that cosine is the most effective metric is not robust to ground-truth completeness.","tokens_in":11280,"feed_emoji":"🔍","tokens_out":13235,"duration_ms":131272,"temperature":0.7,"pith_summary":"The paper asks whether vector embeddings of page images—not just their OCR text—can help people discover relevant content in digitized book collections. Using a vision-language model (ColPali) to represent each page as a multi-vector embedding, the authors issue 75 natural-language queries and compare four distance measures against a manually curated ground truth. They report that cosine similarity retrieves the most relevant pages, with the highest scores on precision, recall, F1, average precision, and mean reciprocal rank. They also find that the system surfaces relevant pages the ground truth missed, which suggests label-based evaluation understates the discovery potential of multimodal vector retrieval.","feed_headline":"Cosine similarity is the top distance measure for page-image retrieval","feed_subtitle":"A vision-language model reads page images, and cosine similarity surfaces both relevant diagrams and text.","key_machinery":"ColPali, a vision-language model that divides each page image into 1,030 visual patches and projects them into a shared 128-dimensional space, producing a multi-vector (1030×128) representation per page. The retrieval mechanism is late-interaction scoring (introduced by ColBERT): each query token scores against every page patch, and max-sum pooling aggregates these scores into a page relevance value. This multi-vector, late-interaction setup is what lets the authors isolate the effect of the four distance functions, since the embeddings and query set are identical across the four Qdrant collections.","core_discovery":"The central empirical claim is that cosine similarity is the most effective distance measure for ColPali-style multi-vector page embeddings in a digital-library setting. With the embedding model, query set, and index held constant, only the similarity function changes across four Qdrant collections; under that controlled comparison, cosine achieves the highest Precision@5 (0.514), Recall@5 (0.281), F1@5 (0.353), Average Precision (0.238), and MRR (0.801) on the full 75-query benchmark, and it also leads on the multi-page and conceptual query subsets. The authors further report that for 18 of 75 queries the system retrieves high-scoring pages outside the gold-standard set, with 72% of those p","pith_inferences":["Because cosine's Precision@5 is 0.514, roughly half of the top five results are not relevant; the limiting component may be the max-sum aggregation rather than the embedding model, so a learned re-ranking step could be the next improvement to test.","The unlabeled-but-relevant hits (18 of 75 queries) suggest a concrete discovery feature: a 'surprise' view that shows users high-similarity pages outside the known relevant set, since 72% of such pages were validated as relevant.","The ground-truth incompleteness cuts both ways: expanding the gold set with the missed relevant pages could widen, shrink, or even reverse the cosine-vs-dot-product margin (0.514 vs 0.496 in Precision@5), so the authors' planned ground-truth strengthening is the decisive test."],"forward_implications":["Adopt cosine similarity as the default distance metric for ColPali/Qdrant page-image retrieval in visually dense collections; it beat the other three metrics on every benchmark.","A single natural-language query can satisfy multi-modal information needs: the top-five results often include both a diagram and its explanatory text.","Top-k evaluation against a manually curated gold standard understates the system's usefulness, since 18 of 75 queries produced relevant pages the annotators had not labeled.","Late interaction means page embeddings are computed once and stored, so the operational bottleneck in a large collection shifts to query-embedding concurrency—flagged in the paper as future work."],"supporting_citations":[{"why":"Supplies the ColPali vision-language model that converts each page image into a multi-vector embedding of 1,030 patches.","marker":"[7]"},{"why":"Supplies PaliGemma-3B, the underlying VLM that extracts patch features and projects them into 128 dimensions.","marker":"[3]"},{"why":"Introduces late-interaction scoring (ColBERT), the retrieval mechanism used to compute query–page relevance scores.","marker":"[10]"},{"why":"Provides the Qdrant vector database that stores page embeddings and implements the four distance functions compared.","marker":"[14]"}],"fun_headline_variants":["Cosine similarity wins for multimodal page retrieval","Cosine similarity beats other measures in VLM search","Best distance measure for image+text retrieval: cosine","Cosine similarity tops benchmark for page-image retrieval"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The comparison of distance measures rests on the manually curated ground truth for the 75 queries being both complete and unbiased; if many relevant pages are missing or mislabeled, every reported metric difference is open to question.","fun_headline_variants_meta":{"raw":{"variants":["Cosine similarity wins for multimodal page retrieval","Cosine similarity beats other measures in VLM search","Best distance measure for image+text retrieval: cosine","Cosine similarity tops benchmark for page-image retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000408,"raw_usage":{"total_tokens":1943,"prompt_tokens":723,"completion_tokens":1220,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":1163}},"tokens_in":467,"tokens_out":1220,"duration_ms":12205,"temperature":1.0,"reasoning_tokens":1163,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T21:00:19.168717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 75-query benchmark with an expanded ground truth that includes the 18 relevant-but-unlabeled pages found in Experiment IV plus any additional pages a second, independent annotator marks as relevant. If cosine similarity's Precision@5 no longer beats dot product's by a meaningful margin, the claim that cosine is the most effective metric is not robust to ground-truth completeness.","supporting_citations":[],"review_version":1}