{"id":"7f68fe89-f044-4eb8-904d-9f25069d9174","arxiv_id":"2506.04997","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Semantic clustering of patch embeddings at the final model layer, plus fine-tuning, preserves 94.6 to 98.2% of retrieval performance while reducing stored embedding memory to 1.8 to 11.8% of original.","lead":"This paper tests ways to shrink the memory used by ColPali and ColQwen2, two systems that search documents by image patches. It finds that merging similar patch embeddings, rather than pruning them, keeps most retrieval quality while cutting memory to as little as 2.8% of the original.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Compression claim rests on query-independent cluster redundancy, but evidence is limited to 5 synthetic queries per page; a held-out real-query evaluation is needed before accepting the reported retention.","rationale":"The reader's weakest assumption matches my main concern: the merging method's success depends on patch embeddings being redundant in a query-independent way, yet the evidence for this redundancy comes only from a small set of synthetic queries. The paper itself shows that different queries activate nearly disjoint patches, which creates a tension with the averaging operation used by semantic clustering. Fine-tuning can mitigate this tension only for query distributions seen during training, and the paper does not test a topic-disjoint held-out query distribution. I do not see a fatal internal flaw; the experiments are extensive and the reported gains are plausible on the evaluated benchmarks. But the central claim of practical storage reduction with minimal retrieval loss is not fully established for unseen queries without such a test. The reader's CONDITIONAL verdict is therefore appropriate, and my concern does not move it.","tokens_in":13830,"tokens_out":7408,"duration_ms":95911,"concrete_test":"On a sample of pages from ViDoRE, compute Section 4.3's activated-patch overlap and r_norm statistics using the official test queries instead of the five synthetic queries, and compute the average per-token max-sim preservation ratio s_merged/s_original before fine-tuning. Then fine-tune Light-ColQwen2 on a topic-disjoint query set (e.g., ArxivQA/Shift only) and evaluate on DocVQA/InfoVQA. If the preservation ratio is below 0.95, or if NDCG@5 retention drops by more than 3 points relative to same-topic evaluation, the query-independent redundancy assumption and the reported retention figures are not supported for unseen queries.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5's merging method reduces each page to N'_p prototypes by clustering embeddings and averaging within clusters. For a query token e_q, the merged score is max_k (avg_{i in C_k} e_i)^T e_q, which is <= max_i e_i^T e_q; equality holds only when every member of the winning cluster has the same dot product with e_q. The paper's support for this query-independent condition is Section 4.3 and Appendix A.2, which use five synthetic queries per page generated by Qwen2-VL-7B. Those same queries show near-disjoint activated patches across queries (Fig. 3a) and only about 14 patches above 0.95 normalized response per page; neither observation establishes that the specific clusters produced by semantic clustering are safe for unseen queries. Because fine-tuning (Section 5.2) is performed on a fixed query distribution, it can compensate for averaging only on query patterns it has seen; no experiment measures retention on a held-out query distribution that is topic-disjoint from training. If real queries activate different patch subsets, the 98.2% and 94.6% figures may not transfer, and the central storage-reduction claim is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies storage reduction for the patch-level embeddings produced by ColPali/ColQwen2 visual document retrievers. It compares two token-reduction families: pruning and merging. On pruning, it reports that random pruning outperforms score- and attention-based strategies, but all pruning methods degrade substantially at aggressive ratios. On merging, it evaluates three dimensions—merging approach (1D/2D spatial pooling vs. semantic clustering), fine-tuning applicability, and merging location (pre-encoder to post-projector)—and finds that late-stage semantic clustering combined with fine-tuning works best. The resulting method, Light-ColPali/ColQwen2, is reported to retain 98.2% of NDCG@5 at 11.8% of original memory and 94.6% at 2.8% memory, averaged over nine datasets from ViDoRE, VisRAG, and MMLongBench-Doc.","tokens_in":14076,"tokens_out":9577,"duration_ms":110560,"significance":"If the reported retention figures hold, this is a practical 9x–35x compression of document embedding storage with small retrieval loss, and the paper provides a systematic comparison of pruning versus merging under the VDR setting. The strengths are the broad evaluation across nine datasets, the combination of training-free and fine-tuned results, and the honest reporting of time costs. The main weakness is that the central assumption—that patch embeddings are redundant in a query-independent way—is supported only by an analysis based on five synthetic queries per page, and no experiment tests transfer to held-out, topic-disjoint queries. The lack of error bars further weakens the precision of the headline numbers.","major_comments":[{"comment":"The central claim that merging via semantic clustering preserves retrieval for arbitrary unseen queries is not supported by the evidence presented. For a merged cluster C_k, the score for a query token e_q is max_k (avg_{i in C_k} e_i)^T e_q, which is bounded above by max_i e_i^T e_q; equality holds only if all members of the winning cluster have identical dot products with e_q. The redundancy analysis in Section 4.3 shows that about 14.2 patches per page have normalized response above 0.95 for a given query, but this does not establish that cosine-similarity clusters are stable across queries. In fact, Figure 3(a) shows that activated patches for two synthetic queries are nearly disjoint, and the synthetic queries themselves are generated by Qwen2-VL-7B. Fine-tuning in Section 5.2 is performed on the ColPali training distribution, so it may compensate for information loss only on query patterns similar to training. No experiment measures retention on a held-out, topic-disjoint query distribution. Please add such an evaluation or provide a direct analysis of within-cluster response-potential variance across diverse query sets; without this, the 98.2% and 94.6% figures are not yet established as a general property of merging.","section":"Section 4.3 / Appendix A.2"},{"comment":"All reported NDCG@5 values are single-run point estimates with no error bars or significance tests. The differences between merging factors (e.g., 99.0% at factor 4, 98.2% at factor 9, 96.3% at factor 25 for Light-ColQwen2) are small, and without variance estimates it is unclear whether these differences are meaningful. Please report means and standard deviations over at least three independent fine-tuning runs, or conduct paired significance tests (e.g., bootstrap or paired t-test) for the key comparisons against ColQwen2 and ColPali.","section":"Tables 2 and 4"},{"comment":"The fine-tuning procedure is underspecified with respect to the clustering module. It is not stated whether the cluster assignments for the merged document embeddings are recomputed at each training step as the model weights change, or fixed using the initial model's embeddings. This matters because if clusters are fixed, the model is trained against a representation that becomes stale as embeddings drift; if recomputed, the training objective changes during optimization. Please clarify and justify the choice, as it directly affects the interpretation of the fine-tuning gains in Figure 6 and the reproducibility of the method.","section":"Section 5.2"},{"comment":"The conclusion that pruning is 'inherently unsuitable' for VDR is based on experiments on only two datasets (DocVQA and InfoVQA) and on synthetic queries generated by Qwen2-VL-7B. While Table 2 extends random pruning to nine datasets, the comparison of the three pruning strategies and the analysis of query-dependent activation are limited to these two datasets. This strong negative claim should either be supported on the full benchmark suite or tempered to a claim about the tested conditions.","section":"Section 4.2"}],"minor_comments":[{"comment":"The definition of Overlap@R is not given in the caption or text; please state how overlap is computed and what the dashed diagonal represents.","section":"Figure 3(a)"},{"comment":"Please specify the linkage criterion (e.g., average, Ward) and the distance metric used in the hierarchical clustering procedure.","section":"Section 5.1"},{"comment":"The 'Average' column appears to report mean NDCG@5, but the caption also mentions relative performance; please clarify what the average column contains and how the relative performance percentages are derived.","section":"Table 2"},{"comment":"The 'ColPali+Pruning' baseline uses random pruning, but this is not stated in the table or its caption; please state it explicitly.","section":"Table 2 / Section 6"},{"comment":"There are several spacing and formatting typos, such as 'to-ken pruningandtoken merging' in the abstract and 'ColPali/-ColQwen2' in the introduction; please proofread.","section":"Abstract / entire manuscript"},{"comment":"The Limitations section does not mention the potential issue that the merging strategy's effectiveness may depend on query distribution; please add a discussion of this limitation, particularly in light of the synthetic-query-based analysis.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"This is a useful empirical study with a simple and likely effective method. The main gaps are the lack of evidence for query-independent redundancy, the absence of error bars, and an underspecified fine-tuning protocol. I would be willing to reconsider after the authors add a topic-disjoint held-out evaluation, variance estimates, and detailed training specifications."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Yubo and colleagues have written a useful empirical paper. The headline result—semantic clustering of patch embeddings at the final projection layer, followed by fine-tuning on merged representations—cuts storage by 9x to 35x while keeping 94-98% of NDCG@5 across nine datasets. That's a practical win, and the paper earns it by testing merging approach, location, and fine-tuning separately. The finding that random pruning beats score- and attention-based pruning is genuinely surprising and the explanation (patch activation is query-dependent, embeddings are groupable) is convincing.\n\nThe paper is honest about its debts: token pooling comes from Clavié et al. and Wu's blog, and the authors say so. The new contribution is the systematic study and the fine-tuned operating points. That is enough for a solid paper.\n\nSoft spots, in rough order of importance. First, no error bars or repeated runs anywhere. With single-run NDCG differences of 1-2%, some of the fine-grained claims (e.g., merging after projector beats before by 0.4%) could be noise. Second, the pruning analysis rests on only two datasets (DocVQA and InfoVQA) and on five synthetic queries per page. The synthetic queries are used to demonstrate why pruning fails; they are not part of the merging method itself. So the stress-test worry about held-out query distributions is real but not disqualifying: the merging method is unsupervised and the main results are evaluated on real benchmark queries. Still, a topic-disjoint held-out set would strengthen the generalization claim. Third, the dataset selection is a bit selective—some saturated ViDoRE datasets and overlapping VisRAG datasets are excluded, which is defensible but should be reported more prominently. Fourth, no code released, which is a shame for an empirical paper.\n\nThe fine-tuning details are reasonable (LoRA, 5 epochs, 130k queries), and the time-cost table is a nice touch. The limitations section is candid about scope.\n\nOverall, this is a competent, useful paper with a clear negative result and a practical positive recipe. The central claim holds up on the evidence presented. I'd like to see error bars and code before publication, but this deserves a serious peer review rather than a desk reject.","headline":"Solid empirical study on compressing ColPali-style patch embeddings; merging at the last layer with fine-tuning works, and the negative result on pruning is the most interesting part.","tokens_in":14621,"tokens_out":2792,"would_cite":true,"duration_ms":32531,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Token merging, not pruning, compresses ColPali/ColQwen2 indexes 9x-35x while retaining roughly 95-98% of retrieval performance.","keywords":["visual document retrieval","ColPali","ColQwen2","token merging","token pruning","patch-level embeddings","storage efficiency","late interaction"],"falsifier":"Compare original ColQwen2 and Light-ColQwen2 at merging factor 9 on a held-out set of real, non-synthetic queries, focusing on text-dense pages such as DocVQA. If a query subset shows a large drop in retrieval of the ground-truth page relative to the 98.2% average—equivalently, if the top-activating patches for those queries are consistently merged away—the query-independent redundancy assumption fails.","tokens_in":13682,"feed_emoji":"🗜️","tokens_out":9682,"duration_ms":101986,"temperature":0.7,"pith_summary":"This paper asks whether the large memory footprint of ColPali/ColQwen2—one 128-dimensional embedding per image patch per page—can be cut without sacrificing retrieval quality. It finds that token pruning cannot: the patches a query activates are almost disjoint across different queries, so attention- and score-based pruning underperform random dropping and collapse under aggressive ratios. Token merging works because page patches are highly redundant and clusterable. The proposed Light-ColPali/ColQwen2 clusters patch embeddings by cosine similarity at the final projection layer, averages each cluster into one prototype, and fine-tunes the retriever on the merged representations. It retains 98.2% of NDCG@5 with 11.8% of the original embedding memory, and 94.6% at 2.8% memory.","feed_headline":"Merge, don't prune: 9x smaller ColPali index, 98.2% retrieval","feed_subtitle":"Patch embeddings are query-dependent, so pruning fails; clustering similar patches at the final layer keeps retrieval scores high.","key_machinery":"The central object is the MaxSim relevance score $s(q,p)=\\sum_j \\max_i e_i^p{}^\\top e_j^q$, which sums, over query tokens, the best matching patch embedding per page. The carrying mechanism is semantic clustering: hierarchical agglomerative clustering of the patch embeddings' cosine similarities, applied after the final projection layer where dimension is lowest and therefore clustering is most effective, with each cluster replaced by its mean vector. Fine-tuning the retriever with these merged embeddings during both training and inference is what recovers most of the performance lost by training-free merging; the merging factor $N_p/N'_p$ sets the storage reduction.","core_discovery":"The central claim is that the stored patch-level embeddings of ColPali/ColQwen2 are substantially redundant, and that redundancy can be exploited by merging rather than pruning. The evidence has three parts: response-potential distributions are clustered, with on average 36.9 patches per page above normalized 0.9 and 14.2 above 0.95; activated patches are query-dependent, so pruning must guess which patches to keep and random dropping wins only because it does not systematically delete whole clusters; and semantic clustering at the post-projector location, followed by fine-tuning, preserves MaxSim-based relevance. Across nine ViDoRE, VisRAG, and MMLongBench-Doc datasets, Light-ColPali/ColQwen2 keeps 99.0% of NDCG@5 at merging factor 4, 98.2% at factor 9, and 94.6% at factor 49, at which point its memory is comparable to single-vector DSE baselines.","pith_inferences":["The query-independent redundancy assumption implies a page-adaptive merging factor: pages with high information density should merge less, and a cheap density estimate could pick the factor per page at index time; the paper notes adaptivity as open future work.","The same late-stage semantic-clustering recipe is a candidate for text-based multi-vector retrievers, where word embeddings may be at least as clusterable as visual patches.","Because the merging module runs offline on stored embeddings, it can be applied post hoc to an already-built ColPali/ColQwen2 index without retraining; the gap to the fine-tuned version then measures how much compression is available for free.","The paper's redundancy evidence rests on five synthetic queries per page; a stress test with human queries across new domains would show whether the 94-98% retention transfers outside the nine benchmark datasets."],"forward_implications":["Light-ColQwen2 keeps 99.0% of NDCG@5 at a merging factor of 4 (25.5% memory) and 98.2% at factor 9 (11.8% memory), so deployment can choose a Pareto point along the reduction curve.","At merging factor 49, the compressed retriever stores 1.8x (Qwen2) or 0.9x (PaliGemma) the memory of a single-embedding DSE retriever while still beating DSE in absolute NDCG@5.","Training-free semantic clustering at factor 9 already retains roughly 97.5% average performance, meaning the merging recipe works even without fine-tuning existing indexes.","Token pruning never reaches competitive retention: at a 0.9-0.95 pruning ratio the best strategy keeps only 58-88% of the original score, so pruning is not a viable route to order-of-magnitude reduction.","Fine-tuning recovers 61% of the performance drop at merging factor 25 and 67% at factor 49, so the marginal cost of extreme compression is mostly recoverable with training."],"supporting_citations":[{"why":"Supplies ColPali/ColQwen2, the retriever whose patch embeddings are compressed, and the MaxSim relevance scoring.","marker":"Faysse et al., 2025"},{"why":"Origin of the token-pooling and merging approaches (1D/2D spatial pooling and semantic clustering) evaluated here.","marker":"Clavié et al., 2024"},{"why":"DSE page-level embedding retrievers, the single-vector memory baseline and a performance comparison point.","marker":"Ma et al., 2024a"},{"why":"Prior blog extending token pooling to ColPali; the proposed method is positioned as the fine-tuned successor.","marker":"Wu, 2024"},{"why":"Provides InfoVQA, a core dataset for the pruning comparison and evaluation.","marker":"Mathew et al., 2021"},{"why":"Provides DocVQA, the other core dataset for the pruning comparison and evaluation.","marker":"Mathew et al., 2020"},{"why":"VisRAG benchmark, source of ChartQA and SlideVQA evaluation sets.","marker":"Yu et al., 2024"},{"why":"MMLongBench-Doc long-context benchmark used for evaluation.","marker":"Ma et al., 2024b"},{"why":"Supplies the attention-oriented pruning strategy used as a baseline baseline for the pruning experiments.","marker":"Chen et al., 2024"}],"fun_headline_variants":["Merging beats pruning for compact ColPali embeddings","Light-ColPali: 11.8% memory, 98.2% retrieval","Cluster patches instead of dropping them for VDR","98.2% retrieval at 11.8% memory via token merging"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Patch-level embeddings are redundant in a query-independent way, so clustering them by cosine similarity and averaging within clusters preserves MaxSim relevance for arbitrary unseen queries; the paper's redundancy evidence uses only five synthetic queries per page, and real queries could activate different patch subsets.","fun_headline_variants_meta":{"raw":{"variants":["Merging beats pruning for compact ColPali embeddings","Light-ColPali: 11.8% memory, 98.2% retrieval","Cluster patches instead of dropping them for VDR","98.2% retrieval at 11.8% memory via token merging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1228,"prompt_tokens":955,"completion_tokens":273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":197}},"tokens_in":571,"tokens_out":273,"duration_ms":3180,"temperature":1.0,"reasoning_tokens":197,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:28:25.642487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare original ColQwen2 and Light-ColQwen2 at merging factor 9 on a held-out set of real, non-synthetic queries, focusing on text-dense pages such as DocVQA. If a query subset shows a large drop in retrieval of the ground-truth page relative to the 98.2% average—equivalently, if the top-activating patches for those queries are consistently merged away—the query-independent redundancy assumption fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies ColPali/ColQwen2, the retriever whose patch embeddings are compressed, and the MaxSim relevance scoring."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides InfoVQA, a core dataset for the pruning comparison and evaluation."},{"cited_title":"Manmatha, and C","cited_arxiv_id":null,"evidence_quote":"Provides DocVQA, the other core dataset for the pruning comparison and evaluation."}],"review_version":1}