{"id":"3ad0a5e4-afac-4782-94d7-a494e0775f72","arxiv_id":"2505.24782","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new benchmark (ConTEB) and training method (InSeNT) show that context-aware chunk embeddings greatly improve retrieval on context-dependent queries, with minimal computational overhead.","lead":"This paper presents ConTEB, a new benchmark for testing whether retrieval models can use the whole document when indexing chunks, and shows current models fail on context-dependent queries. It also introduces InSeNT, a lightweight contrastive training method that substantially improves context-aware chunk embeddings without increasing indexing time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified disjointness between ConTEB's MLDR/NarrativeQA/SQuAD sets and the released InSeNT training data threatens the headline in-domain gains; an overlap check on the released artifacts would settle it.","rationale":"Reader's weakest assumption is exactly this split-disjointness concern, and the manuscript's own wording makes it concrete rather than hypothetical: Section 3.3 restricts MLDR and NarrativeQA to training splits but not SQuAD, and Section 3.2 does not say that ConTEB evaluation uses held-out splits for any of these datasets. The central claim depends on ConTEB averages that include the three in-domain sets, so a nonzero overlap would directly inflate the strongest quantified result (52.0 to 75.6 average nDCG@10). I do not think this concern by itself justifies rejection: the OOD controlled tasks (Football: 19.1 to 63.9; Insurance: 12.4 to 100.0; Geography: 56.2 to 90.7) are manually or semi-synthetically constructed and are not part of the training corpus, so the core mechanism has independent support. A separate internal tension exists in the abstract's 'without sacrificing base model performance' clause versus the NanoBEIR column (ModernBERT 63.2 to 60.4; ModernColBERT 67.7 to 59.2), but that is a wording and claim-calibration issue rather than the primary load-bearing risk to the benchmark results. The reader's CONDITIONAL verdict remains appropriate: require the overlap check on released artifacts, and require the authors to either state the exact splits used or report overlap-free numbers.","tokens_in":15806,"tokens_out":10803,"duration_ms":120062,"concrete_test":"Download the released ConTEB evaluation sets and the released InSeNT training data. Compute exact or near-duplicate overlap by (a) stable document IDs and chunk hashes for MLDR, NarrativeQA, and SQuAD, and (b) normalized query text after lowercasing and whitespace normalization. If any overlap exists, rerun Table 2 on the overlap-free subset; if an in-domain nDCG@10 changes by more than about 1 point, the headline ConTEB improvement is partly an artifact of split leakage. If the overlap is zero, the concern is resolved and the in-domain results stand as reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The evaluation assumes that the ConTEB sets for MLDR, NarrativeQA, and SQuAD are disjoint from the data used to train InSeNT. Section 3.3 states that the released training set 'includes the training splits of MLDR and NarrativeQA' and then says 'We also concatenate SQuAD chunks from the same Wikipedia article' without restricting SQuAD to a training split. Table 1 lists MLDR, NarrativeQA, and SQuAD as in-domain ConTEB tasks, and the paper never states which original splits are used for evaluation. If any documents or query-chunk pairs appear in both training and evaluation, the in-domain gains (MLDR 78.4 to 88.7, SQuAD 73.4 to 80.9, NarrativeQA 77.9 to 81.3) are inflated, directly supporting the abstract's broad claim about 'significantly improves retrieval quality on ConTEB.' The controlled out-of-domain tasks (Football, Geography, Insurance) are less exposed, but they cannot validate the in-domain columns that dominate the headline average. This is a data-provenance gap, not an attack on the method's design; the open-source release makes it checkable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ConTEB, a benchmark for evaluating whether text embedding models exploit document-wide context when encoding chunks, and InSeNT, a contrastive post-training objective that combines in-batch and in-sequence negatives with late chunking. On ConTEB, InSeNT substantially improves nDCG@10 over the late-chunking baseline for ModernBERT and ModernColBERT, with the largest gains on the controlled Football/Geography/Insurance tasks. The paper also reports ablations on the in-sequence weight, chunking robustness, and corpus scaling, and it releases the benchmark, models, and training data.","tokens_in":16053,"tokens_out":6018,"duration_ms":63378,"significance":"If the results hold, the paper makes a useful contribution: it provides a public benchmark for context-aware chunk retrieval and a cheap post-training procedure that improves contextualization without the inference cost of LLM-based reformulation. The release of all artifacts, the clear ablation isolating the role of in-sequence vs. in-batch negatives, and the demonstration that concatenated short documents do not substitute for organic long documents are concrete strengths. The main claims rest on two points that need verification: split disjointness between training and evaluation, and the statement that InSeNT improves retrieval 'without sacrificing base model performance.'","major_comments":[{"comment":"The paper does not establish that the ConTEB evaluation sets for MLDR, NarrativeQA, and SQuAD are disjoint from the InSeNT training data. Section 3.3 states that the released training set includes the training splits of MLDR and NarrativeQA and that SQuAD chunks are concatenated, without restricting SQuAD to a non-training split; Table 1 lists MLDR, NarrativeQA, and SQuAD among the in-domain ConTEB tasks; and the evaluation section never states which original splits are used. If any evaluation documents or query-chunk pairs appear in the training data, the reported in-domain gains (e.g., MLDR 78.4 to 88.7, SQuAD 73.4 to 80.9, NarrativeQA 77.9 to 81.3) are inflated and would propagate to the headline average. Please state the evaluation split for each dataset explicitly and, since the artifacts are released, provide an overlap check between training and evaluation instances.","section":"§3.3 and Table 1"},{"comment":"The abstract claims InSeNT improves retrieval 'without sacrificing base model performance,' but the NanoBEIR column of Table 2 shows ModernBERT Large dropping from 63.2 to 60.4 and ModernColBERT dropping from 67.7 to 59.2 after InSeNT training. The 'Short-Context Performance' paragraph in §5 asserts that 'careful hyperparameter tuning enables our best model to maintain strong performance,' but no confidence intervals or significance tests are given, so a 2.8-point drop (and an 8.5-point drop for the ColBERT variant) cannot be claimed to be non-significant. Please either soften the claim to acknowledge a measurable NanoBEIR degradation or provide statistical evidence that the difference is within noise.","section":"Abstract and §5, Table 2"}],"minor_comments":[{"comment":"References are duplicated: Warner et al. 2024a and 2024b are the same paper, Nussbaum et al. 2024 and 2025 share the same title, and Zhong et al. 2025a and 2025b are duplicates. Please consolidate these entries.","section":"References"},{"comment":"The 'Sabotage' step uses LLM-reformulated paragraphs and LLM-generated questions for the controlled tasks, so the conclusion that 'state-of-the-art embedding models struggle' should be framed as a property of the constructed benchmark rather than a universal deficiency of non-contextual encoders.","section":"§3.2"},{"comment":"The x-axis label 'Max Chunk Length' with a 'No chunking' category is unclear; please specify how the 'no chunking' condition is constructed and how relative nDCG@10 is computed.","section":"Figure 4 (left)"},{"comment":"The observation that naively feeding documents longer than 8,192 tokens to ModernBERT outperforms the sliding-window approach is surprising; please provide implementation details (e.g., truncation, position embeddings) given that the base model was trained with an 8,192-token context.","section":"Appendix C.3"}],"recommendation":"major_revision","confidential_remarks":"The data-provenance question is the most important risk. If the authors can demonstrate that the ConTEB evaluation is disjoint from the InSeNT training data, I would view the contribution favorably. The paper should also be asked to make the NanoBEIR claim consistent with Table 2. I see no evidence of misconduct; the issue is a missing statement that is easily checkable via the released artifacts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. The core contribution is real: ConTEB fills an actual gap in retrieval evaluation, and InSeNT is a simple extension of Late Chunking that clearly works. The paper is honest about its design, open-sources the artifacts, and includes a nice control (ModernBERT trained on the same data, flattening the batch) that shows the gains are not just from the data. The ablations with concatenated short documents and the robustness-to-chunking experiments are also well done. I would not be surprised if ConTEB becomes a standard test for contextual retrieval, and InSeNT is cheap enough to be a practical fix.\n\nThe soft spots are real but mostly addressable. The biggest is data provenance: Section 3.3 says the training set includes MLDR and NarrativeQA training splits, plus SQuAD chunks, and Table 1 lists those same three as in-domain ConTEB tasks. The paper never states which splits are used for ConTEB evaluation. If any of the same query-chunk pairs appear in both, the in-domain gains are inflated. The controlled out-of-domain tasks (Football, Geography, Insurance) are less exposed and still show large gains, so the central claim about context-dependence does not collapse, but the abstract's broad 'significantly improves retrieval quality on ConTEB' should not rest on unverified splits. This is easy to fix with a one-line statement or a quick overlap check on the released data.\n\nAlso, the 'without sacrificing base model performance' claim is contradicted by their own NanoBEIR numbers: ModernBERT+InSeNT drops from 63.2 to 60.4, and ModernColBERT+InSeNT drops from 67.7 to 59.2. They acknowledge the LI degradation and float future fixes, but the abstract should be toned down.\n\nMinor: there are no error bars or significance tests, so it is hard to know how much of the smaller differences (e.g., SQuAD 73.4 to 80.9) is noise. And the controlled tasks are deliberately 'sabotaged' to make context necessary, so the message is more 'here is what contextual models can do when context matters' than 'state-of-the-art models unexpectedly fail on natural data.'\n\nOverall, this is a serious, well-executed paper that deserves referee time. I would send it to peer review with a request to clarify split disjointness, add error bars or at least reruns on a few seeds, and soften the no-sacrifice claim. The out-of-domain gains are the strongest evidence, so I would ask the authors to foreground those in the abstract.","headline":"The paper delivers a genuinely useful benchmark for context-dependent retrieval and a cheap, effective training method, but the 'no sacrifice' claim is overstated and the training/evaluation split overlap needs to be checked.","tokens_in":763,"tokens_out":691,"would_cite":true,"duration_ms":27254,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chunk embeddings that can see the whole document beat independent chunk embeddings on context-dependent retrieval, and a cheap contrastive post-training method, InSeNT, delivers most of the gain.","keywords":["contextual retrieval","document embeddings","late chunking","contrastive learning","in-sequence negatives","retrieval benchmark","long-context embeddings","RAG chunking"],"falsifier":"Re-run the ConTEB evaluation on MLDR, NarrativeQA, and SQuAD using held-out splits that are provably disjoint from the InSeNT training set and compare nDCG@10; if the in-domain gains shrink to near the non-contextual baseline, the headline improvements come from train/evaluation overlap rather than from context propagation.","tokens_in":15606,"feed_emoji":"🔎","tokens_out":6694,"duration_ms":65462,"temperature":0.7,"pith_summary":"Retrieval systems typically embed each chunk of a long document in isolation, so a sentence like “He became emperor in 1804” carries no link to Napoléon. This paper argues that this missing document-wide context is a major, under-measured failure mode, and introduces ConTEB, a benchmark built from controlled and real-world tasks where answering requires context beyond the chunk itself. On ConTEB, strong off-the-shelf embedders score far below methods that propagate context, and the paper's InSeNT post-training method, applied on top of late chunking, lifts average nDCG@10 from 52.0 to 75.6 for ModernBERT-Large while leaving short-context performance essentially intact. The intended contribution is a cheap, open way to make chunk embeddings context-aware without relying on expensive LLM reformulation at indexing time.","feed_headline":"Context-aware training lifts retrieval scores from 52 to 75.6","feed_subtitle":"A new benchmark shows chunk embeddings miss document context; InSeNT fixes it at almost no cost.","key_machinery":"Late chunking, extended by the InSeNT objective. Late chunking runs the entire document through the encoder in one forward pass and average-pools token representations over each chunk's span, letting chunk embeddings see the rest of the document before pooling. InSeNT trains this with a weighted InfoNCE loss that combines standard in-batch negatives from other documents with “in-sequence” negatives drawn from other chunks of the same document; $\\lambda_{\\text{seq}}$ controls the mix, and the same loss extends to late-interaction models by substituting ColBERT's MaxSim for the dot product. The in-sequence term is what prevents all chunks of a document from collapsing into one indistinguishable representation.","core_discovery":"The central claim is that chunk-level retrieval embeddings should be computed with access to the whole source document, and that this can be trained for efficiently. The paper shows that standard single-vector and late-interaction embedders fail on ConTEB's context-dependent tasks (e.g., Insurance nDCG@10 of 12.4 for ModernBERT-Large vs. 100.0 after InSeNT), while untrained late chunking alone already helps on controlled tasks. InSeNT adds same-document chunks as hard negatives to a contrastive objective, so each chunk absorbs document-level information while staying distinguishable from its neighbors. The authors report that this post-training costs about one H100 GPU hour per epoch, preserves NanoBEIR non-contextual performance, and makes retrieval more robust to poor chunking and larger corpora.","pith_inferences":["Editorial extension: if chunk embeddings reliably inherit document context, then chunking strategy itself becomes less critical, and systems could use much smaller chunks to improve downstream answer granularity.","Editorial extension: the in-sequence negative principle should transfer to decoder-based long-context embedders, where whole-book encoding could make context propagation even stronger; the paper names this as future work.","Editorial extension: one testable consequence is that InSeNT should help most on corpora full of templated documents that differ by a few key fields; the robustness-to-corpus-size result suggests this, but it is not measured directly."],"forward_implications":["On ConTEB's controlled tasks, context-aware retrieval can jump from near-random to near-perfect: Insurance goes from 12.4 to 100.0 nDCG@10 for ModernBERT-Large with InSeNT.","InSeNT-trained chunk embeddings tolerate much smaller, less self-contained chunks without the usual retrieval collapse.","Contextual chunk embeddings degrade more gracefully as the corpus grows with many similar documents, a common industrial condition.","The method adds almost no indexing cost (about 15 ms per document for ModernBERT-Large + InSeNT) and avoids the 120x slowdown of LLM-based contextual retrieval.","Short-context retrieval on NanoBEIR is not sacrificed when hyperparameters are tuned ($\\lambda_{\\text{seq}} = 0.1$)."],"supporting_citations":[{"why":"Introduces late chunking, the full-document pooling mechanism InSeNT trains on top of.","marker":"Günther et al., 2024"},{"why":"Supplies the InfoNCE contrastive objective that InSeNT extends with in-sequence negatives.","marker":"Oord et al., 2018"},{"why":"Establishes dense passage retrieval contrastive training, the framework behind the paper's batch construction.","marker":"Karpukhin et al., 2020"},{"why":"Provides ModernBERT, the long-context encoder that makes full-document forward passes feasible.","marker":"Warner et al., 2024a"},{"why":"Releases modernbert-embed-large, the base single-vector model for the main experiments.","marker":"Chaffin, 2025b"},{"why":"Releases GTE-ModernColBERT, the base late-interaction model for the LI variant.","marker":"Chaffin, 2025a"},{"why":"Defines ColBERT late interaction and MaxSim, used for the LI version of InSeNT.","marker":"Khattab and Zaharia, 2020"},{"why":"Is the LLM-based contextual retrieval baseline that InSeNT matches at far lower indexing cost.","marker":"Anthropic, 2024"},{"why":"Provides BEIR/NanoBEIR, used to check non-regression on standard non-contextual retrieval tasks.","marker":"Thakur et al., 2021"},{"why":"Supplies SQuAD, used both as a ConTEB task and as part of the InSeNT training data.","marker":"Rajpurkar et al., 2016"}],"fun_headline_variants":["ConTEB exposes context blindness; InSeNT training closes the gap","Chunk embeddings learn document context via InSeNT post-training","New benchmark, new fix: InSeNT adds context to chunk embeddings","Context-aware chunk vectors: InSeNT boosts retrieval on ConTEB","Teaching chunks to read their document: InSeNT retrieval boost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ConTEB evaluation sets for MLDR, NarrativeQA, and SQuAD are disjoint from the training splits used to train InSeNT; the paper does not state which splits ConTEB evaluates, so if any query-chunk pair appears in both, the reported in-domain gains would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["ConTEB exposes context blindness; InSeNT training closes the gap","Chunk embeddings learn document context via InSeNT post-training","New benchmark, new fix: InSeNT adds context to chunk embeddings","Context-aware chunk vectors: InSeNT boosts retrieval on ConTEB","Teaching chunks to read their document: InSeNT retrieval boost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000354,"raw_usage":{"total_tokens":1893,"prompt_tokens":879,"completion_tokens":1014,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":936}},"tokens_in":495,"tokens_out":1014,"duration_ms":11170,"temperature":1.0,"reasoning_tokens":936,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:13:15.358926+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ConTEB evaluation on MLDR, NarrativeQA, and SQuAD using held-out splits that are provably disjoint from the InSeNT training set and compare nDCG@10; if the in-domain gains shrink to near the non-contextual baseline, the headline improvements come from train/evaluation overlap rather than from context propagation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the LLM-based contextual retrieval baseline that InSeNT matches at far lower indexing cost."}],"review_version":1}