{"id":"c404fcf2-28bf-47ed-b18c-2ab623e64e4c","arxiv_id":"2607.05927","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"A contextual multimodal document retrieval benchmark (CMDR-Bench) and embedding model (CMDR-Embed) that jointly encodes multiple document pages and splits them into page-level representations, trained with a context-aware contrastive objective, outperforming non-contextual baselines by 13–16 nDCG@5.","lead":"This paper introduces a new benchmark and method for retrieving pages from multi-page documents where the answer requires reasoning across pages, not just matching keywords on a single page. The method jointly encodes groups of pages to capture cross-page context, then splits them into page-level embeddings, improving retrieval by 16+ points over independent-page encoding.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Benchmark quality control removed queries solvable by non-contextual retrievers—including ColPali, a Table 2 baseline—creating selection bias that directly inflates the contextual vs. non-contextual gap.","rationale":"The reader's CONDITIONAL verdict is appropriate. The concern about train-eval distribution alignment is real and well-directed, and I've sharpened it by identifying the specific mechanism: the benchmark filtering uses the same models as baselines, creating selection bias that inflates the gap. However, the paper's controlled comparison (same training data for both contextual and non-contextual models) does isolate the architectural contribution to some degree, and the ablations (Table 3) provide useful evidence about which components matter. The paper also honestly reports that CMDR-Embed underperforms on non-contextual retrieval (Table 5, ViDoRe), which is consistent with a real architectural tradeoff rather than pure benchmark gaming. The CONDITIONAL verdict captures the right level of uncertainty: the directional claim (contextual encoding helps for contextual retrieval) is well-supported, but the magnitude is benchmark-specific and the three-way alignment between filtering, training data, and architecture prevents strong confidence in the 16.2-point figure as a generalizable result. No verdict change needed—the reader already flagged the right concerns at the right severity level.","tokens_in":35772,"tokens_out":4115,"duration_ms":273893,"concrete_test":"Construct a held-out evaluation set of contextual retrieval queries WITHOUT applying the Recall@1=1.0 filtering step—i.e., include queries that non-contextual models can solve via direct matching. Evaluate both CMDR-Embed_Qwen and ColQwen+Finetuned on this unfiltered set. If the gap remains approximately 16 points, the architectural claim is robust. If the gap shrinks substantially (e.g., below 8 points), the filtering process is driving the measured advantage and the headline magnitude is an artifact of benchmark construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 describes a filtering process where three non-contextual retrievers (ColPali [17], VLM2Vec [23], NV-Embed [26]) were used to compute average Recall@1 for each query. Queries scoring 1.0 were removed as 'trivial' (16% of queries discarded). ColPali is one of the baselines in Table 2. This means the benchmark was explicitly constructed to exclude queries that ColPali and other non-contextual models can solve, and then those same models are evaluated on the remaining harder subset. The 16.2-point gap between CMDR-Embed_Qwen and ColQwen+Finetuned is measured on a test set selected to disadvantage non-contextual models by construction. This is not merely a representativeness concern—it is selection on the dependent variable. The filtering ensures the benchmark contains only queries where independent page encoding is insufficient, which is precisely the capability that CMDR-Embed's joint encoding provides. Combined with CMDR-Synth training data generated through prompts that mirror the benchmark's four-category structure (Supplementary B, step 3 prompt: 'the query should appear to be about Image 1, but the correct answer is found only in Image 2'), there is a three-way alignment between benchmark filtering, training data distribution, and the proposed architecture. The architectural advantage is likely real—joint encoding should help on contextual tasks—but the magnitude (16.2 points) is difficult to interpret because the evaluation setup is optimized to showcase exactly this advantage. The reader identified the train-eval alignment but did not fully develop how the filtering process uses the same models that serve as baselines, which is the more precise mechanism of bias.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper introduces CMDR-Bench, a benchmark for contextual multimodal document retrieval where queries require cross-page reasoning to identify the relevant page, and CMDR-Embed, a model that jointly encodes multiple pages via a sliding window and derives page-level embeddings from a shared contextual representation. The benchmark comprises 800 human-annotated queries across four categories (Text Completion, Coreference Resolution, Structured Understanding, Multi-hop Reasoning) over 255 long documents. CMDR-Embed is trained with a contextual multimodal contrastive learning objective (CMCL) that balances context-aware hard negatives (in-chunk and in-document) with standard in-batch negatives. Experiments show CMDR-Embed outperforming finetuned non-contextual baselines by 13.5–16.2 nDCG@5 points, with ablations isolating the contribution of each component.","tokens_in":36626,"tokens_out":1361,"duration_ms":280176,"significance":"The paper addresses a genuine gap: existing multimodal document retrieval benchmarks do not evaluate indirect, context-dependent retrieval across pages. The chunk-then-split architecture is a natural and well-motivated adaptation of late-chunking ideas to the visual document setting. The experimental design is thorough: ablations (Table 3) isolate CMCL components, the same training data is used for finetuned baselines to control for data effects, efficiency is analyzed (Fig. 7, Table 4), and generalization to non-contextual retrieval is tested (Table 5). The inclusion of Recall@5 results (Table C.3), a reranker baseline (Table C.4), and OCR-quality analysis (Table C.5) strengthens the evaluation. The qualitative examples (Fig. 11) and error analysis (Figs. C.9–C.11) provide useful insight into failure modes.","major_comments":[{"comment":"§3.2, Data quality control: The benchmark filtering removes queries where three non-contextual retrievers (including ColPali, a Table 2 baseline) achieve average Recall@1 = 1.0, discarding 16% of queries as 'trivial.' This creates selection on the dependent variable: the test set is explicitly constructed to exclude queries solvable by non-contextual models, and those same models are then evaluated on the remaining harder subset. The 16.2-point gap between CMDR-Embed_Qwen and ColQwen+Finetuned is thus measured on a set optimized to disadvantage exactly the baseline it is compared against. The paper should either (a) report results on the full unfiltered set as well, or (b) explicitly acknowledge this as a scope limitation and reframe the magnitude claims accordingly. The current framing ('significantly outperforms') is defensible but the magnitude is difficult to interpret without the un","section":null},{"comment":"§5.1 and Supplementary B: The training data CMDR-Synth is generated by LLM prompts that explicitly target the same four query categories used in the benchmark (step 3 prompt includes '{query type}'). This creates a train-eval distribution alignment that could inflate measured gains. While the paper controls for training data by finetuning baselines on CMDR-Synth (which is good practice), the synthetic data generation pipeline mirrors the benchmark's task structure, so both contextual and non-contextual models trained on CMDR-Synth benefit from this alignment. The paper should discuss this alignment explicitly and, if possible, report results on an out-of-distribution contextual retrieval set (e.g., DAPR or ConTEB) to assess transfer.","section":null},{"comment":"Table 2, CR column: CMDR-Embed_Qwen achieves only 38.4 nDCG@5 on Coreference Resolution, a mere 3.6-point improvement over ColQwen+Finetuned (34.8), compared to 18.8 and 16.7 on TC and SU respectively. Since CR is one of the four categories that defines the benchmark and motivates the contextual encoding architecture, this weak result on a core category warrants discussion. The paper notes that CR and MR are 'particularly challenging' (§5.2) but does not analyze why the contextual architecture provides minimal benefit on CR specifically. This affects the central claim that joint encoding captures cross-page contextual relationships.","section":null}],"minor_comments":[{"comment":"§5.2: The claim of 'minimal computational overhead' is supported by Fig. 7, but the text does not state the specific indexing time numbers. Including the key numbers (e.g., 'X ms/page for ColPali+Finetune vs. Y ms/page for CMDR-Embed') in the text would make the claim more concrete.","section":null},{"comment":"Table 5: CMDR-Embed_Qwen drops from 88.9 to 85.5 on ViDoRe non-contextual retrieval. The multitask variant recovers to 89.2 but drops on contextual (56.6→56.1). The trade-off should be discussed more explicitly in the text rather than only in the table.","section":null},{"comment":"§4.1: The notation for chunk indexing uses s(t-1)+1 but does not clarify whether pages are 0-indexed or 1-indexed. Clarify.","section":null},{"comment":"Figure 6: The x-axis labels '>3' for context distance are ambiguous. Consider using explicit bin labels like '1', '2', '3', '4+'.","section":null},{"comment":"§3.2: The annotators are described as 'authors, who hold PhDs in computer science.' The number of annotators and inter-annotator agreement statistics are not reported. Adding these would strengthen the quality claim.","section":null},{"comment":"Table C.4: Applying the reranker to CMDR-Embed_Pali actually hurts performance (49.8→41.0). The explanation that 'conventional rerankers are not explicitly trained to capture inter-page dependencies' is plausible but speculative; consider softening.","section":null},{"comment":"Reference [29] (Qwen3-VL Embedding) is dated 2026. Confirm this is correct and not a typo for 2025.","section":null}],"recommendation":"major_revision","confidential_remarks":"The selection-on-dependent-variable concern (filtering out queries solvable by baselines, then evaluating those baselines) is the most serious issue. It is fixable: the authors could report results on the pre-filtering set, or at minimum reframe the magnitude claims. The train-eval category alignment is a lesser but real concern. The architecture and CMCL loss are sound contributions; the benchmark fills a genuine gap. I would lean toward acceptance if the authors address the filtering concern transparently."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a thorough and constructive review. The referee correctly identifies three important concerns: (1) selection bias from filtering trivial queries using non-contextual retrievers, (2) train-eval distribution alignment between CMDR-Synth and CMDR-Bench, and (3) weak CR performance relative to other categories. We address each below and commit to revisions for all three.","responses":[{"response":"The referee raises a valid concern. We acknowledge that the filtering process introduces a form of selection bias: by removing queries solvable by non-contextual retrievers, the benchmark is explicitly scoped to contextual retrieval, and the magnitude of improvement over non-contextual baselines is measured on this harder subset. We agree this should be made transparent. In revision, we will: (a) report results on the full unfiltered set (before removal of the 16% trivial queries) so readers can see the complete picture, (b) explicitly reframe the scope of our magnitude claims, noting that the reported gains apply specifically to queries requiring contextual reasoning rather than to all possible retrieval queries over these documents, and (c) clarify in the benchmark description that CMDR-Bench is designed to evaluate contextual retrieval specifically, not general document retrieval. We note that the filtering was conducted with human oversight—annotators reviewed flagged queries and made revision or removal decisions—but the referee's point about interpretability of the magnitude is well taken.","revision_made":"yes","referee_comment":"§3.2 Data quality control: filtering removes queries where three non-contextual retrievers achieve Recall@1=1.0, creating selection on the dependent variable. The 16.2-point gap is measured on a set optimized to disadvantage baselines. Should report results on full unfiltered set or acknowledge as scope limitation."},{"response":"The referee correctly identifies that the synthetic training data generation pipeline mirrors the benchmark's task structure, since the query generation prompt (step 3) includes the query type. This is a legitimate concern. We will address it in two ways. First, we will add an explicit discussion of this alignment in the paper, acknowledging it as a potential limitation on the interpretability of absolute gains. Second, regarding OOD evaluation: DAPR and ConTEB are text-only, single-page benchmarks and thus not directly applicable to our multimodal, multi-page setting. However, we can report results on ViDoRe (Table 5 already provides partial evidence of generalization to non-contextual retrieval). For a contextual OOD test, we note that no existing multimodal benchmark evaluates contextual cross-page retrieval in the same formulation as CMDR-Bench, which is precisely the gap our benchmark fills. We will state this limitation honestly and note that constructing an independent contextual multimodal retrieval benchmark is an important direction for future work. We will also emphasize that the controlled comparison (finetuning baselines on the same CMDR-Synth data) isolates the architectural contribution of contextual encoding from the data contribution, even if both benefit from the train-eval alignment.","revision_made":"partial","referee_comment":"§5.1 and Supplementary B: CMDR-Synth generation prompts explicitly target the same four query categories as the benchmark, creating train-eval distribution alignment that could inflate gains. Should discuss this alignment and report OOD results (e.g., DAPR or ConTEB)."},{"response":"The referee is correct that the CR category shows the smallest gain from contextual encoding, and we agree this warrants deeper analysis. We will add a dedicated discussion in Section 5.2. Our analysis of CR errors reveals two main factors. First, coreference resolution often requires resolving references to entities or terms that appear on pages far from the relevant page—sometimes beyond the chunk window (w=4, s=2). When the context page lies outside the jointly encoded chunk, CMDR-Embed cannot leverage it, and performance degrades to approximately the non-contextual baseline level. This is consistent with Figure 6, which shows that performance drops as the context distance increases, particularly for in-chunk negatives. Second, CR queries frequently involve pronouns or abbreviations that are semantically generic (e.g., 'this,' 'the aforementioned'), making the query-page similarity signal weak regardless of contextualization. The contextual architecture helps when the context page provides disambiguating content that shifts the embedding, but when the referent is a generic token, the late-interaction mechanism has limited signal to exploit. We will add this analysis and note that extending the chunk window and incorporating bidirectional attention (as discussed in our Limitations section) are promising directions for improving CR specifically.","revision_made":"yes","referee_comment":"Table 2, CR column: CMDR-Embed_Qwen achieves only 38.4 nDCG@5 on Coreference Resolution, a 3.6-point improvement over ColQwen+Finetuned (34.8), compared to 18.8 and 16.7 on TC and SU. Weak result on a core category warrants discussion of why contextual architecture provides minimal benefit on CR."}],"tokens_in":35770,"tokens_out":1042,"duration_ms":146586,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Short version: the paper introduces a genuinely new task (contextual multimodal page retrieval), a human-annotated benchmark, and a chunk-then-split encoding method with a dedicated contrastive loss. The core architectural claim — joint encoding beats independent encoding — is real and well-controlled. The magnitude of the gain is harder to interpret because the benchmark filtering, training data, and architecture all align to favor contextual retrieval. Worth a serious referee; the filtering concern is the main thing to press on in review, not a reason to reject outright.","headline":"Solid benchmark + method for contextual multimodal retrieval; main concern is train-eval-filter three-way alignment inflating the gap.","tokens_in":36600,"tokens_out":172,"would_cite":false,"duration_ms":147265,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Jointly encoding document pages cuts retrieval errors by 16 points","keywords":[],"falsifier":"If a non-contextual retriever, given the same backbone and training data, could match CMDR-Embed's performance by simply increasing the number or diversity of hard negatives during training, the contextual encoding architecture would not be the causal source of the improvement. Alternatively, if the benchmark's query categories were expanded to include contextual retrieval scenarios not anticipated by the four defined types, and CMDR-Embed's advantage disappeared on those new categories, the gains would be specific to the benchmark's task definition rather than evidence for the general value  ","tokens_in":35981,"feed_emoji":"📄","tokens_out":923,"duration_ms":154565,"temperature":0.7,"pith_summary":"This paper claims that encoding multiple document pages together, rather than one at a time, produces substantially better page-level retrieval for queries that require cross-page reasoning. The authors introduce CMDR-Bench, a benchmark of 800 human-annotated queries over multi-page documents (averaging 183.5 pages) where the correct page can only be identified by leveraging context from other pages. They propose CMDR-Embed, which uses a chunk-then-split strategy: consecutive pages are encoded jointly by a vision-language model, then the shared representation is separated back into page-level embeddings. A companion training objective, CMCL, balances contextual modeling against page-level discriminability by introducing two types of hard negatives drawn from the same document. The central empirical result is that CMDR-Embed outperforms the same backbone finetuned on the same data by 16.2 nDCG@5 points (56.6 vs. 40.4), isolating the contextual encoding architecture as the source of improvement rather than training data or model scale.","feed_headline":"Jointly encoding document pages cuts retrieval errors by 16 points","feed_subtitle":"Pages encoded together, then split apart, capture cross-page context that independent page encoders miss entirely.","key_machinery":"CMDR-Embed uses a chunk-then-split encoding pipeline: a sliding window groups consecutive document pages into chunks of size w with stride s, each chunk is processed jointly by a vision-language model, and the resulting token-level representations are separated back into per-page embeddings. Page-level similarity to a query is computed via Late Interaction, which sums the maximum dot product between each query token vector and the page's token vectors. Training uses CMCL loss, a weighted combination of a standard in-batch InfoNCE loss and a context-aware loss that treats other pages from the same chunk (in-chunk negatives) and other pages from the same document but outside the current chunk ","core_discovery":"The paper's central discovery is that the independent-page encoding assumption used by current multimodal document retrievers is a substantial performance bottleneck for contextual queries. When pages are encoded jointly in overlapping chunks and then split, the resulting page-level embeddings carry cross-page contextual information that enables retrieval of pages whose relevance is not explicitly stated in the query but must be inferred from surrounding pages. The 16.2-point gain over a matched non-contextual baseline, combined with ablations showing that both the contextual encoding and the CMCL contrastive objective contribute independently, establishes that the architecture of encoding, ","pith_inferences":[],"forward_implications":["Document retrieval systems that currently encode pages independently could adopt joint encoding with modest computational overhead, since the paper reports indexing latency comparable to the backbone model.","The benchmark's four query categories (text completion, coreference resolution, structured understanding, multi-hop reasoning) provide a diagnostic framework for identifying specific failure modes in existing retrievers, particularly in coreference and multi-hop scenarios where all models scored lowest.","The finding that training data gains saturate around 20k-40k instances while the contextual architecture continues to outperform suggests that architectural design, not data scaling, is the current limiting factor for contextual document retrieval.","The multitask learning result showing that contextual and non-contextual retrieval can coexist without degradation suggests future systems can adopt joint encoding as a default without sacrificing standard retrieval performance."],"fun_headline_variants":["Joint page encoding beats independent retrieval by 16.2 points","Shared context embeddings cut cross-page retrieval errors by 16 points","Encoding pages together then splitting them improves retrieval 16 points","Contextual multimodal embeddings outperform independent encoders by 16 points"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The benchmark's 800 human-annotated queries across four author-defined categories are assumed to adequately represent the full space of contextual retrieval needs in real-world documents. The synthetic training data is generated to match exactly these four categories using LLM prompts that mirror the benchmark's task structure, creating a tight train-eval distribution alignment that could inflate the measured gains relative to what would be observed on naturally occurring检索  ","fun_headline_variants_meta":{"raw":{"variants":["Joint page encoding beats independent retrieval by 16.2 points","Shared context embeddings cut cross-page retrieval errors by 16 points","Encoding pages together then splitting them improves retrieval 16 points","Contextual multimodal embeddings outperform independent encoders by 16 points"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":554,"prompt_tokens":497,"completion_tokens":57,"prompt_tokens_details":null},"tokens_in":497,"tokens_out":57,"duration_ms":59214,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T20:10:06.837939+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a non-contextual retriever, given the same backbone and training data, could match CMDR-Embed's performance by simply increasing the number or diversity of hard negatives during training, the contextual encoding architecture would not be the causal source of the improvement. Alternatively, if the benchmark's query categories were expanded to include contextual retrieval scenarios not anticipated by the four defined types, and CMDR-Embed's advantage disappeared on those new categories, the gains would be specific to the benchmark's task definition rather than evidence for the general value","supporting_citations":[],"review_version":1}