{"id":"0356722d-70be-422a-aa0c-48e6e7f7df6a","arxiv_id":"2608.02969","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"MarginMerge compresses ColPali and ColQwen multi-vector indexes through coverage-aware anchors and learned representative synthesis, preserving 97-99% of nDCG@5 at 5-10% vector retention.","lead":"This paper introduces MarginMerge, a way to compress multi-vector visual document retrieval indexes to 5-10% of their original size while keeping roughly 97-99% of retrieval quality. It argues that compression should preserve the diverse page regions different queries might match, rather than just the most salient patches, and this principle guides the method's design.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot transfer is not actually tested: the global prototype bank used for coverage scoring and anchor selection includes training-side query tokens from the three 'unseen' evaluation datasets (App. A.1), contaminating the main zero-shot results.","rationale":"The reader identified the prototype bank's representativeness as the weakest assumption, which is related but not the same as my concern. The reader's framing treats it as a general distribution-shift risk, whereas Appendix A.1 reveals a concrete, avoidable contamination: the 'unseen' evaluation datasets contribute their own training-side query tokens to the prototype bank. This means the main zero-shot evaluation does not test what it claims to test. The leave-one-dataset-out study partially addresses the issue, but it covers only the three training datasets and shows a failure on Flickr relative to geometric merging, so it does not rescue the claim for DocVQA/InfoVQA/TAT-DQA. This is not an internal inconsistency in the method, but a mismatch between the stated contribution and the evaluation protocol. The fix is straightforward and computational: exclude target training queries from the prototype bank and rerun. If performance holds, the claim is salvaged; if not, the paper's transfer contribution and the reported averages need revision. The reader's CONDITIONAL verdict remains appropriate, but the condition should explicitly include this rerun; the concern is more specific than the reader's stated weakest assumption, hence 'partial' agreement.","tokens_in":14870,"tokens_out":4942,"duration_ms":52132,"concrete_test":"Reconstruct the prototype bank Z using training-side query tokens from only ArxivQA, TabFQuad, and Flickr (excluding DocVQA, InfoVQA, and TAT-DQA entirely), then rerun MarginMerge compression and MaxSim retrieval for those three datasets at 5% and 10% retention, using the same frozen backbones and seeds. Compare the resulting nDCG@5 values to those in Table 1. If any of the three datasets drops by more than 0.005, the zero-shot claim is not supported and the headline averages must be recomputed; if the numbers are essentially unchanged, the contamination is negligible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim includes that 'the same model transfers to unseen datasets' (Abstract, Contributions). The main evidence is Table 1, where DocVQA, InfoVQA, and TAT-DQA are described as zero-shot because the weighting network is not trained on them. However, Appendix A.1 states: 'The global prototype bank uses training side query tokens from all six retrieval datasets.' The prototype bank Z enters the coverage score C_ti(d) in Eq. 4 and the anchor objective F_d(A) in Eq. 5, which determine the clusters and thus the synthesized representatives. Therefore the compression for these 'unseen' datasets is informed by query statistics from those same datasets, making the evaluation transductive rather than zero-shot. The leave-one-dataset-out study (Table S2) does remove the target from prototype construction, but it covers only ArxivQA, TabFQuad, and Flickr (the three training datasets), and on Flickr MarginMerge still trails geometric merging (0.6840 vs 0.7536). Consequently, the main evidence for cross-dataset transfer is not clean, and the reported nDCG@5 values on DocVQA, InfoVQA, and TAT-DQA — and the averaged 97–99% preservation figures — may be inflated by target-domain query directions in the prototype bank. This is a load-bearing issue because the 'generalizes across datasets' contribution rests on these zero-shot numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MarginMerge, an index-time compression method for frozen ColPali/ColQwen multi-vector retrievers. It selects coverage-aware anchors from a prototype bank of query directions, clusters document patch vectors, and learns a lightweight shared network to synthesize one representative per cluster, keeping the standard MaxSim retrieval interface. Experiments on six datasets and two backbones report nDCG@5 at 5% and 10% vector retention, claiming 97-99% preservation of full-index quality while reducing stored vectors by 90-95%, plus transfer to unseen datasets and retention ratios. The supplementary material contains per-seed results, ablations, a leave-one-dataset-out study, proofs, and a code archive.","tokens_in":15229,"tokens_out":6959,"duration_ms":76570,"significance":"If the empirical claims hold, this is a practically useful and well-scoped contribution: compression happens once at index time, retrieval uses the unmodified MaxSim engine, and the reported numbers are internally consistent. The paper is strengthened by per-seed reporting, a documented leakage audit, a fixed configuration across datasets, and a clean conditional proof of the submodularity result in Supplement C.1. However, the zero-shot transfer evidence is weakened by the construction of the prototype bank, and the flagship coverage-aware anchor component is not the best variant in the full system. These issues bear directly on the paper's main contributions and need to be addressed before publication.","major_comments":[{"comment":"The paper describes DocVQA, InfoVQA, and TAT-DQA as zero-shot/unseen and states that 'The zero shot datasets are excluded from all fitting stages.' Appendix A.1, however, states that 'The global prototype bank uses training side query tokens from all six retrieval datasets.' The prototype bank Z enters the coverage score C_ti in Eq. (4) and the anchor objective F_d(A) in Eq. (5), and it also defines the prototype-relevance features consumed by the learned network. The compressed representations of documents in the three 'unseen' datasets are therefore constructed with query-direction information drawn from those same datasets (their training-side splits). This makes the evaluation transductive rather than zero-shot and could inflate the reported DocVQA, InfoVQA, and TAT-DQA numbers. The leave-one-dataset-out study in Table S2 does remove the target from prototype construction, but it covers only ArxivQA, TabFQuad, and Flickr, and on Flickr MarginMerge remains below geometric merging (0.6840 vs 0.7536). Please rerun the zero-shot evaluation with a prototype bank built only from the three training datasets, or restrict the claim to 'unseen by the weighting network' and clearly label the current protocol as transductive.","section":"Appendix A.1 and Experimental Setup (with Eqs. 4-5)"},{"comment":"In the full configuration, k-center anchors with the learned full objective achieve a higher average nDCG@5 than the reported coverage-aware anchors: 0.875 vs 0.865 at 5% retention and 0.886 vs 0.883 at 10% retention on ColQwen2.5. Since the reported MarginMerge configuration uses coverage-aware anchors, the paper's headline numbers are not the best configuration of its own method. More importantly, the claim that coverage-aware anchor selection is a core component of the final method is not supported by this table: the benefit of coverage-aware anchors appears only when the representative is fixed (e.g., 'Retained anchor' and 'Response centroid' rows), while learned synthesis makes the anchor rule largely irrelevant or slightly harmful. The paper should either report the k-center variant as the main configuration, with a clear explanation of why the precommitted coverage-aware anchor rule was retained, or substantially revise the claim that 'coverage-aware anchors' are responsible for the method's success.","section":"Table 4"}],"minor_comments":[{"comment":"The flip-rate columns are not labeled with the method to which each number belongs; the pairs such as .0033/.0048 and .0621/.1095 are ambiguous as printed. Please add explicit column headers such as 'MarginMerge flip rate' and 'Geometric merging flip rate'.","section":"Table S1"},{"comment":"The symbol '≈' is used in Table 1 without a definition in the table caption; the definition appears only in the Experimental Setup text. A brief note in the caption would improve readability.","section":"Experimental Setup"},{"comment":"The phrase 'The same model transfers to unseen datasets' should be qualified in the abstract, because the prototype bank uses training-side query tokens from all six datasets and only the weighting network is unseen by the three evaluation datasets.","section":"Abstract and Contributions"},{"comment":"The two spatial coordinates are set to zero in all reported runs, so the network effectively does not use spatial features. This should be stated in the main text as well, since the feature description in the main method implies spatial information is available.","section":"Appendix A.4"},{"comment":"The claim that ranking flips are reduced 'by approximately 41% on average' does not obviously follow from the per-dataset flip rates in Table S1, which give an average reduction of about 36% if the first value in each pair is MarginMerge and the second is geometric merging. Please clarify the aggregation or the interpretation of the pair ordering.","section":"Stability and Transfer"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical paper with a reproducible code archive and careful ablation structure. The main concern is terminological and conceptual: the 'zero-shot' datasets are not truly unseen because their training-side query tokens enter the prototype bank used by the compression pipeline. This is fixable by re-running without target-side tokens or by rewriting the claim, and it should be fixed before acceptance. The Table 4 anchor discrepancy is also important because it undercuts the 'coverage-aware anchors' contribution; the authors should either change the reported configuration or provide a clear justification for the precommitted choice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MarginMerge is a genuinely useful compression method for multi-vector visual retrievers. The core result—a shared network synthesizing one representative per cluster under margin distillation, preserving 97–99% nDCG@5 at 5–10% retention on six datasets—looks solid and internally consistent. The ablations are careful: per-seed numbers, a leave-one-out study, a controlled factorial, and explicit statements about what is and isn't claimed. The finding that learned synthesis beats patch selection, and that diversity beats winner coverage, is a real contribution.\n\nWhat's new is the specific assembly: coverage-aware anchor selection, prototype relevance scoring, learned cluster weighting, and margin distillation. The paper also does something rare: it admits that the flagship anchor choice isn't actually best. Table 4 shows k-center anchors with the full objective reach 0.875 vs 0.865 for the reported coverage-aware configuration at 5% on ColQwen. That inconsistency is minor because the main gain comes from learned synthesis, but it should be flagged.\n\nThe load-bearing problem is the zero-shot claim. Appendix A.1 states that the global prototype bank uses training-side query tokens from all six retrieval datasets, including DocVQA, InfoVQA, and TAT-DQA. The bank enters coverage scoring and anchor selection, so the compression for those 'unseen' datasets is informed by their query statistics. That makes the evaluation transductive, not zero-shot. The leave-one-out study removes the target from the bank, but it only covers the three training datasets, and on Flickr MarginMerge still trails geometric merging. So the clean cross-dataset transfer evidence is thin. The abstract and contributions overstate what is shown.\n\nOther soft spots are smaller. The code is archived but not released; given the complexity, a reviewer should request it. Indexing cost is 1.7s per TAT-DQA document, acknowledged but worth quantifying at scale. The redundancy analysis is explicitly observational, which is fine.\n\nNone of this breaks the central claim: at fixed retention, the method improves over matched baselines, and the 97–99% preservation figure is likely correct for the reported setup. But the generalization narrative needs rewriting. I'd send it to peer review. The contamination is fixable by re-labeling the evaluation or, better, running a proper leave-one-out across all six datasets and reporting those numbers. If they hold, the transfer claim becomes credible.","headline":"The compression results are probably real, but the 'zero-shot' transfer claim is contaminated by the prototype bank.","tokens_in":15758,"tokens_out":2046,"would_cite":true,"duration_ms":22984,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MarginMerge compresses multi-vector visual document retrievers by preserving query-relevant coverage, cutting stored vectors by 90–95% while retaining 97–99% of average nDCG@5 across six datasets and two backbones.","keywords":["multi-vector retrieval","visual document retrieval","index compression","query-relevant coverage","MaxSim late interaction","ColPali","ColQwen","learned representative synthesis"],"falsifier":"Build a target corpus whose evaluation queries are shifted so that the winning document patches lie in regions never activated by any training-query token, then run MarginMerge at 5% vector retention; if its nDCG@5 falls to or below uniform random retention (the baseline it currently beats), the prototype-bank premise is falsified.","tokens_in":14701,"feed_emoji":"🗜️","tokens_out":9613,"duration_ms":80389,"temperature":0.7,"pith_summary":"Multi-vector visual document retrievers such as ColPali and ColQwen store hundreds of patch embeddings per page, which gives strong retrieval but large indexes and costly late-interaction scoring. The paper argues that aggressive compression should not pick the individually most salient patches; it should preserve query-relevant coverage, the diverse page regions that could become the best MaxSim match for some query. The authors introduce MarginMerge, which selects coverage-aware anchors, clusters the original patch embeddings, and trains a small shared network to synthesize one representative vector per cluster at index time, leaving the online MaxSim scorer unchanged. Across six datasets and two backbones, MarginMerge reports the highest matched query-agnostic average nDCG@5 at 5% and 10% vector retention, preserving 97–99% of the full-index average while storing only 5–10% of the vectors, and it transfers to unseen datasets and retention ratios without retraining. If correct, the work reframes index compression as a set-coverage problem and explains why dense rendered pages compress more easily than natural images.","feed_headline":"MarginMerge keeps 97–99% of retrieval quality at 5–10% of vectors","feed_subtitle":"Because it preserves the regions queries actually match, 90% smaller indexes still rank correctly.","key_machinery":"The load-bearing object is the coverage-aware anchor selection objective $F_d(A) = \\sum_{t=1}^M \\omega_t \\max_{a \\in A} C_{ta}$, where $C_{ti} = \\exp(-(b_t(d) - z_t^\\top v_i)/\\tau_c)$ is the coverage that patch $i$ provides for query prototype $z_t$, and $b_t(d) = \\max_i z_t^\\top v_i$ is the prototype's best match in the document. Since each $C_{ta}$ is monotone submodular in the anchor set, greedy anchor selection carries the standard $(1-1/e)$ approximation guarantee. The anchors define a partition of the document patches; a small shared network $g_\\theta$ produces per-patch log-weights that combine anchor similarity, prototype relevance, and a learned residual, and the cluster representative is $r_c = \\mathrm{Norm}(\\sum_{i \\in C_c} \\alpha_i v_i)$, a normalized convex combination of original patch vectors. A ranking-margin distillation loss then matches the compressed margins $m^\\theta_j = \\hat S(q,d^+) - \\hat S(q,d^-_j)$ to the full-index margins $m^{\\mathrm{full}}_j$, weighted toward pairs near the ranking boundary, so the compression preserves document ordering rather than absolute scores. This machinery is what lets the method stay inside the standard MaxSim interface while changing the representation.","core_discovery":"At its center, the paper claims that compressing a multi-vector document representation is a set-coverage problem rather than a patch-ranking problem. Because MaxSim gives each query token the score of its best-matching document patch, the evidence that must survive compression is the union, over the query distribution, of document regions that can serve as the strongest match; keeping several near-duplicate high-scoring patches is less valuable than representing complementary regions. The paper formalizes this with a prototype bank of query directions, a per-patch coverage score, and a monotone submodular anchor objective that greedy selection optimizes to the $(1-1/e)$ approximation guarantee. MarginMerge clusters the original patches by coverage-aware anchors, then synthesizes one representative per cluster as a normalized convex combination of its patches using a 1,057-parameter shared network, trained to match full-index ranking margins. The reported result is that on six datasets over both ColQwen2.5 and ColPali, this preserves 97–99% of mean nDCG@5 while storing 5–10% of the document vectors, with transfer to held-out datasets and retention ratios without retraining.","pith_inferences":["Editorial inference: the coverage principle should apply to any late-interaction retriever, including ColBERT-style text retrieval, where the same MaxSim account predicts that merging clusters of substitutable token embeddings by query direction would beat salience pruning.","Editorial inference: re-estimating the prototype bank on a small sample of target-distribution queries, or updating it at index refresh time, could make the method robust to domain shift without retraining the weighting network.","Editorial inference: the indexing bottleneck is greedy anchor selection, about 1.7 seconds per TAT-DQA document in the reported implementation; submodularity guarantees suggest sampling or lazy evaluation could scale this to billions of pages.","Editorial inference: because representatives are already normalized and cluster-local, quantizing them to bytes should compound with MarginMerge additively; the paper defers byte-level evaluation."],"forward_implications":["At 5–10% vector retention, a frozen ColQwen2.5 or ColPali index retains 97–99% of average nDCG@5, so large visual-document corpora can be served at a fraction of storage and MaxSim cost.","At 5% retention, MarginMerge reduces ranking flips relative to geometric merging by about 41% on average across the six ColQwen2.5 datasets, making the compressed ranking more faithful to the full index.","A single network trained at 5% retention transfers to 10% and 20% retention and to datasets held out from training (DocVQA, InfoVQA, TAT-DQA) without retraining.","Patch selection methods based on predicted importance or salience can underperform uniform random retention, while diversity-aware selection preserves retrieval quality, supporting the coverage-over-salience account.","Compressibility orders as dense rendered documents > scene-text photographs > natural photographs, matching the effective-rank redundancy statistic across nine datasets."],"supporting_citations":[{"why":"Supplies the MaxSim late-interaction scoring operator that the compression problem is defined against.","marker":"(Khattab and Zaharia 2020)"},{"why":"Introduces ColPali, the multi-vector visual document retriever whose patch embeddings MarginMerge compresses.","marker":"(Faysse et al. 2025)"},{"why":"Provides Light-ColPali geometric merging, the strongest matched baseline, and the empirical observation that pruning can underperform random retention.","marker":"(Ma et al. 2025)"},{"why":"Contributes token merging, the origin of the merge-based compression family MarginMerge extends.","marker":"(Bolya et al. 2023)"},{"why":"Provides the Qwen2.5-VL backbone underlying ColQwen2.5, one of the two frozen retrievers evaluated.","marker":"(Bai et al. 2025)"},{"why":"Supplies the DSE single-vector baseline used as the lower-bound control.","marker":"(Ma et al. 2024)"},{"why":"Provides SAP, a structural-anchor pruning baseline used as a diagnostic for the coverage account.","marker":"(Liu et al. 2026)"},{"why":"Supplies HPC-style hierarchical patch compression, included as an upper-reference query-conditioned baseline.","marker":"(Duong and Minh 2025)"}],"fun_headline_variants":["MarginMerge: compress to 10% size, keep 99% retrieval","Coverage-aware compression: 97–99% quality at 5–10% vectors","Coverage matters: shrink index 10x, keep 97% quality","Preserve query coverage, cut vectors 90%, keep retrieval quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the 128 query-prototype directions built from training-query tokens are representative of the query directions the index will actually face; if test queries activate document regions far outside those directions, the coverage scores, anchors, and learned weights lose their grounding.","fun_headline_variants_meta":{"raw":{"variants":["MarginMerge: compress to 10% size, keep 99% retrieval","Coverage-aware compression: 97–99% quality at 5–10% vectors","Coverage matters: shrink index 10x, keep 97% quality","Preserve query coverage, cut vectors 90%, keep retrieval quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00116,"raw_usage":{"total_tokens":4847,"prompt_tokens":1030,"completion_tokens":3817,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":3733}},"tokens_in":646,"tokens_out":3817,"duration_ms":29530,"temperature":1.0,"reasoning_tokens":3733,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:23:45.980811+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a target corpus whose evaluation queries are shifted so that the winning document patches lie in regions never activated by any training-query token, then run MarginMerge at 5% vector retention; if its nDCG@5 falls to or below uniform random retention (the baseline it currently beats), the prototype-bank premise is falsified.","supporting_citations":[{"cited_title":"Token Merging: Your","cited_arxiv_id":null,"evidence_quote":"Contributes token merging, the origin of the merge-based compression family MarginMerge extends."},{"cited_title":"2025 , doi =","cited_arxiv_id":null,"evidence_quote":"Provides the Qwen2.5-VL backbone underlying ColQwen2.5, one of the two frozen retrievers evaluated."},{"cited_title":"Hierarchical Patch Compression for","cited_arxiv_id":null,"evidence_quote":"Supplies HPC-style hierarchical patch compression, included as an upper-reference query-conditioned baseline."}],"review_version":1}