{"id":"f4d1a9c4-d556-4e04-9a68-836fdd541e54","arxiv_id":"2506.00773","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Dynamic chunking plus question-aware chunk selection improves long-context QA, but the headline numbers are partly inflated by choosing hyperparameters on the test benchmarks.","lead":"This paper introduces a method that splits very long documents into variable-length chunks based on semantic similarity, then uses a trained classifier to keep only chunks relevant to a given question before sending them to a large language model. The authors report consistent gains over several long-context baselines on question-answering benchmarks with inputs up to 256,000 tokens.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Chunk-selection classifier is trained on whole-context answerability but applied to individual chunks; without chunk-level validation the central selection mechanism is unproven.","rationale":"I focused on the classifier because the paper's novelty and central claim depend on it: dynamic chunking alone is a modest extension of known semantic-splitting ideas, and the abstract explicitly credits the question-aware classifier with selecting sensitive chunks. The training protocol in Section 3.2 and the appendix examples (Figures 10-12) confirm that positives are full passages labeled answerable, while inference presents chunks; this is a train/test instance-type mismatch, not merely domain shift. The paper's Table 5 only varies training corpora, so it cannot detect this mismatch. The absence of any retrieval-recall or oracle-evidence metric means the reader cannot tell whether the observed QA gains come from selecting the right chunks or from the LLM's robustness to compressed input. A chunk-level label test is cheap and decisive. I did not base the verdict on the test-set hyperparameter sweep, although it is a real secondary concern, because the sweep is openly reported and the long-context results use the same settings; it inflates point estimates but does not threaten the mechanism as directly. The dynamic chunking ablation and classifier-vs-cosine comparison are useful, but the cosine comparison actually underscores the concern because the classifier's advantage is small and sometimes negative. Overall the paper is plausible and the empirical results are suggestive; a CONDITIONAL verdict requiring chunk-level validation is appropriate.","tokens_in":21404,"tokens_out":8368,"duration_ms":82614,"concrete_test":"On held-out passages from AdversarialQA/CoQA/SQuAD, apply the DCS chunker and label each chunk as positive iff it contains the gold answer span. Measure the trained classifier's chunk-level AUC and recall@k with k matching the paper's compression ratio. Additionally retrain the same MLP on these chunk-level labels and compare end-task accuracy on HotpotQA, Musique, and Loogle_SD against the current whole-context-trained classifier. If chunk-level AUC is near chance, or if the chunk-trained classifier changes selection and end-task results materially, the current training objective does not support the claimed chunk-selection mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 trains the question-aware classifier on whole document-question pairs: positives are [C,Q] with the full context, Eq. (8) concatenates all of C, and negatives are unrelated context-question pairs. At inference, however, the classifier scores individual chunks [c_i; Q] to select top-k chunks. These are different instance types: during training the model sees whole passages that contain an answer somewhere; during inference it sees short chunks, most of which do not contain the answer, and for multi-hop questions no single chunk is answerable in isolation. Nothing in the training procedure provides chunk-level labels, so the classifier may learn document-level topical relevance or 'the passage is answerable' rather than localization of evidence. The paper reports no chunk-level retrieval recall or oracle analysis; Table 4 shows the learned classifier is only marginally better than cosine-similarity selection and is worse on Llama3 single-hop (32.90 vs 33.07), so the claimed selection mechanism is not independently confirmed. This is load-bearing because the central claim is that DCS 'trains a question-aware classifier to select sensitive chunks'; if the classifier cannot rank chunks by evidence content, the reported QA gains are not explained by the proposed mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dynamic Chunking and Selection (DCS), a training-light method for long-context reading comprehension. DCS first splits the input into variable-length chunks by computing sentence embeddings and cutting at low-similarity boundaries, then trains a small question-aware classifier on top of features extracted from a frozen LLM (boundary hidden states and attention maps) to score each chunk with respect to the question. The top-scoring chunks are concatenated and fed to the LLM within its context window. The authors evaluate DCS against StreamingLLM, LM-Infinite, InfLLM, and MoICE on 12 QA datasets (single- and multi-hop) using Llama-3-8B-Instruct, Mistral-7B-Instruct, and Vicuna-7B, including versions up to 256k tokens, and report consistent average improvements as well as an ablation and latency analysis.","tokens_in":21664,"tokens_out":8770,"duration_ms":75171,"significance":"If the reported results hold, DCS is a simple and reproducible baseline that extends LLM reading comprehension to very long inputs without architectural changes or heavy training, and the release of code and data is valuable. The dynamic chunking idea is plausible and the comparison across three backbones is broader than in many long-context papers. However, the significance is currently limited by the lack of validation that the learned classifier actually selects evidence chunks, the selection of key hyperparameters on the test benchmarks, and underspecified inference protocols for out-of-window inputs; these issues must be addressed before the claims can be fully trusted.","major_comments":[{"comment":"The question-aware classifier is trained on whole-context question-answerability pairs, where the positive label indicates that the full context contains an answer somewhere, but at inference it is applied to individual chunks (each chunk paired with the question). The manuscript provides no chunk-level validation (e.g., recall of the oracle evidence chunks or an oracle-top-k experiment), so it is not established that the classifier learns to localize evidence rather than to detect topical relevance. Table 4 shows that the trained classifier is actually worse than cosine-similarity selection on Llama-3 single-hop QA (32.90 vs. 33.07) and only slightly better on average (26.85 vs. 25.84). Since the paper's central claim is that DCS 'trains a question-aware classifier to select sensitive chunks,' this lack of direct evidence weakens the attribution of the reported gains to the proposed selection mechanism. Please report chunk-level retrieval metrics and/or retrain the classifier with chunk-level supervision and show consistent gains across all three backbones.","section":"§3.2, Eqs. (8)–(23), Table 4"},{"comment":"The hyperparameters l and α are chosen by evaluating on the same six datasets (NarrativeQA, HotpotQA, 2WikiMQA, MFQA_en, Qasper, Musique) that are part of the main evaluation in Table 1. Selecting hyperparameters on the test benchmarks means that the headline improvements in Table 1 are partly the result of fitting to the evaluation sets. Please use a separate validation split (or a small development set disjoint from the final benchmarks) and report the resulting performance; alternatively, show that the reported gains are stable under a wide range of l and α chosen without access to the test labels.","section":"§5.4.1, Table 2"},{"comment":"The protocol for evaluating inputs longer than the backbone's context window is not specified. The manuscript states that Llama-3-8B-Instruct and Mistral-7B-Instruct have a maximum length of 8K and Vicuna-7B has 4K, but the long-context experiments use inputs of 16k–256k tokens. It is not described how the 'Origin' baseline (or the StreamingLLM/LM-Infinite/InfLLM/MoICE baselines) is applied to such inputs—e.g., whether the context is truncated, split, or processed with the baseline's own mechanism. Without this information, the comparison against 'Origin' is not a well-defined baseline, and the claim in §5.3 that 'baselines suffer from significant performance deterioration' is not supported by Figure 3, which plots only 'Origin' and DCS rather than the named baselines. Please specify the exact inference protocol for all methods on out-of-window inputs and include the actual baseline curves (or tables) for the 64k–256k region.","section":"§4.2, Figure 3"},{"comment":"The caption of Table 1 states that 'The t-test proves that the improvement is statistically significant (p<0.05),' but no details are given about the test procedure, the unit of analysis, the number of samples, or whether the test is paired. Similarly, the average improvements in Tables 4 and 5 are reported without error bars or significance testing. Please provide a precise description of the statistical test and, ideally, report confidence intervals or per-dataset significance levels.","section":"Table 1 caption and §5"}],"minor_comments":[{"comment":"The selection criterion uses ⌊m/α⌋, but the compression ratio is defined as αc in Eq. (22); please use consistent notation.","section":"Eq. (23)"},{"comment":"The entry '44.7921.25' appears to be two numbers (44.79 and 21.25) merged without a space; please fix the formatting.","section":"Table 1"},{"comment":"'Ispossitive' should be 'Is positive' in the example annotations.","section":"Figures 10–12"},{"comment":"The negative sampling strategy S is described only as generating 'semantically uncorrelated pairs'; please specify how these pairs are generated (e.g., random pairing within the same corpus) and whether any filtering is applied.","section":"§3.2"},{"comment":"The main text lists MoICE as a baseline, but the main table only reports Llama-3 results; please clarify in the text where the MoICE results (Mistral and Vicuna) are presented and why the main table omits them.","section":"§4.2"},{"comment":"The complexity statement O(nl) should also account for the cost of encoding each chunk with the backbone LLM when computing classifier features; please state the total computational cost more carefully.","section":"§5.4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible and simple method, and the reported gains are interesting if they hold. My main concerns are the test-set selection of hyperparameters, the lack of direct evidence for the classifier's chunk-selection capability, and the underspecified protocol for long-context baselines. These are fixable with additional experiments and clearer reporting, so I recommend major revision rather than rejection. The authors should also double-check the statistical significance claim and the formatting errors before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a practical compression pipeline with broad empirical coverage, but the chunk-selection part is not actually demonstrated to work as described, and the headline numbers are partly fitted to the test sets. The dynamic chunking alone is the more solid piece.\n\nWhat's new: semantic-similarity chunking is not new (they cite Kamradt), and question-aware chunk selection basically is prompt compression / retrieval, so the novelty is a combination plus an empirical study. The study is substantial: 12 QA datasets, three 7B/8B LLMs, lengths up to 256k. That is real work and it shows a consistent gain over the baselines they picked.\n\nWhat it does well: the dynamic chunking ablation beats fixed chunking, the method is low-cost, and the cross-training-data robustness (AdversarialQA/CoQA/SQuAD) is a nice touch. If the gains hold, DCS is a useful tool for long-document QA and RAG.\n\nThe soft spots are serious but specific. The most load-bearing is the classifier train/inference mismatch. Section 3.2 trains on whole context-question pairs, positive if the full context contains an answer, negative otherwise. At inference it scores each chunk. Those are different instance types. No chunk-level labels are used and no chunk-level retrieval recall is reported. Table 4 shows the learned classifier is only slightly better than cosine-similarity and actually worse on Llama3 single-hop (32.90 vs 33.07). So the paper does not establish that the selection mechanism is what causes the gains. This is not a fatal flaw, but the central claim in the abstract is 'train a question-aware classifier to select sensitive chunks,' and that claim is unproven.\n\nSecond, hyperparameters l and alpha are chosen on the same test sets (Table 2), which makes the reported numbers optimistic. Third, the handling of inputs longer than the LLM context window is not specified for the Origin and baseline models, so the 256k results could involve different input truncation across methods. Fourth, no code snapshot is provided despite the abstract link.\n\nI'd send this to a serious referee. The evaluation breadth and the dynamic chunking result are worth checking, and a reviewer can ask for a chunk-level retrieval evaluation, a retuning protocol on a validation split, and a cleaner baseline description. The paper is honest in its limitations section and not circular. For readers working on long-context QA, it is worth a read, but I would not rely on the reported magnitudes yet.","headline":"Useful compression pipeline with broad empirical coverage, but the chunk-selection mechanism is not actually validated and the headline numbers are partly fitted to the test sets.","tokens_in":22163,"tokens_out":3034,"would_cite":false,"duration_ms":26816,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a two-stage pipeline—semantic-boundary chunking plus a question-aware relevance classifier—lets small LLMs answer questions from contexts up to 256k tokens more accurately than streaming, eviction, or fixed-chunk…","keywords":["long-context reading comprehension","dynamic chunking","question-aware chunk selection","large language models","question answering","semantic similarity","multi-hop QA","context compression"],"falsifier":"A concrete falsifier: on a held-out domain such as legal or medical documents, mark the single chunk that contains the gold answer, let the classifier rank all chunks, and compute whether the evidence chunk appears in the selected top-k. If the selection recall is no better than random for multi-hop questions whose evidence is spread across four or more chunks, the paper's robustness claim is overturned.","tokens_in":21238,"feed_emoji":"🧩","tokens_out":11235,"duration_ms":94890,"temperature":0.7,"pith_summary":"Long-context reading comprehension fails not only because context windows are short, but because fixed-length splitting cuts through sentences and separates the evidence an answer needs. The paper proposes Dynamic Chunking and Selection (DCS): first split a document at places where adjacent sentences are semantically dissimilar, keeping each chunk coherent; then use a small question-aware classifier to keep only the chunks that matter for the question, and feed those to the LLM in original order. DCS is training-light, since only the classifier is trained, and it lets 7-8B instruction-tuned models work with inputs far beyond their native windows. Across twelve single- and multi-hop QA datasets, DCS reports consistent gains over streaming, eviction, and fixed-chunk baselines, with the smallest degradation when context length grows from 16k to 256k tokens.","feed_headline":"Dynamic chunking beats fixed-length cutting for long-context QA","feed_subtitle":"Tested on 12 QA benchmarks, it stays accurate as contexts grow to 256k tokens.","key_machinery":"The central object is the chunk: a variable-length unit of text that preserves sentence integrity. Dynamic chunking builds these units by splitting the document on punctuation into sentences, merging each sentence with its immediate neighbors, encoding the merged segments with a sentence embedding model, and computing cosine similarity between adjacent encoded sentences; the lowest-similarity positions become candidate boundaries, subject to a percentile threshold $\\alpha$, and chunks are then merged again to stay under a length cap $l$. The second object is a question-aware classifier: a three-layer MLP whose input is a distilled feature vector of six pieces—the hidden states of the first and last context and question boundary tokens plus attention-pooled context and question representations taken from the backbone LLM's final layer. Trained with binary cross-entropy on answerable versus unanswerable context-question pairs, the classifier gives each chunk a relevance score $T_i$; the top $\\lfloor m/\\alpha_c \\rfloor$ chunks are concatenated in their original order with the question into the LLM's context window.","core_discovery":"The central claim is that the semantic structure of a document is the right unit for compressing long contexts before question answering. DCS first recovers that structure by encoding sentences and cutting where cosine similarity between neighboring sentences is low, so coherent units—complete sentences and their supporting context—stay together. It then trains a three-layer classifier on question-aware features distilled from the LLM's own final-layer hidden states and attention scores; at inference the classifier scores every chunk for the given question and only the top chunks are passed to the LLM. The paper reports that this configuration beats all compared baselines on most of twelve QA datasets: on the largest 8B backbone it raises average single-hop score from 27.60 to 35.50 and multi-hop score from 24.22 to 29.07, and it holds performance as inputs scale to 256k tokens.","pith_inferences":["Editorial inference: the classifier's relevance scores could be reused as a learned reranker in retrieval pipelines, turning a document-level QA method into a chunk-level retriever without additional training.","Editorial inference: because DCS preserves the original order of kept chunks, part of its gain may come from moving evidence out of the middle of long inputs; a position-shuffling experiment would separate this effect from pure evidence selection.","Editorial inference: the dynamic chunker is unsupervised and model-agnostic, so it may transfer to other long-text tasks such as summarization or code completion, though the paper evaluates only question answering."],"forward_implications":["Contexts up to 256k tokens can be answered by models whose native window is 4k-8k tokens, with less performance loss than streaming or eviction baselines.","Multi-hop questions benefit disproportionately, because keeping each supporting fact inside an intact chunk makes it possible for the LLM to combine evidence rather than receive a sentence fragment.","The method adds almost no training cost: only the small MLP classifier is trained, so it can be dropped onto new backbone LLMs without fine-tuning them.","Ablations show both halves earn their keep: dynamic chunking beats fixed-length chunking, and the learned classifier beats cosine-similarity selection."],"supporting_citations":[{"why":"Supplies the sentence embedding model used to compute similarity between adjacent sentences for dynamic chunk boundaries.","marker":"Reimers and Gurevych, 2019"},{"why":"Provides the semantic-splitting idea that low embedding similarity marks natural boundaries; DCS builds its chunker on it.","marker":"Kamradt, 2023"},{"why":"Supplies the answerable context-question pairs used as training data for the question-aware classifier.","marker":"Bartolo et al., 2020"},{"why":"StreamingLLM is a baseline that DCS must beat on long-context attention.","marker":"Xiao et al., 2023"},{"why":"LM-Infinite is a baseline with attention masks and distance ceilings that DCS is compared against.","marker":"Han et al., 2024"},{"why":"InfLLM is a fixed-chunk memory baseline that DCS compares against and outperforms.","marker":"Xiao et al., 2024"},{"why":"MoICE is an in-context-expert baseline used in the Mistral and Vicuna comparisons.","marker":"Lin et al., 2024"},{"why":"LongBench supplies several of the QA evaluation datasets used to score DCS.","marker":"Bai et al., 2024b"},{"why":"LV-Eval supplies the length-controlled 16k to 256k versions of the benchmarks.","marker":"Yuan et al., 2024"},{"why":"Loogle supplies long-document QA datasets for both single-hop and multi-hop evaluation.","marker":"Li et al., 2023"}],"fun_headline_variants":["Dynamic chunking keeps QA accurate as contexts reach 256k","Chunk at semantic gaps, select by question, read far longer","Question-aware dynamic chunking outperforms fixed slicing","Semantic chunking and selection for ultra-long context QA","Dynamic chunking at semantic boundaries boosts long-context QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a classifier trained on a few open-domain QA datasets can rank chunks by relevance well enough for every new document, question type, and context length; if its relevance concept fails to transfer to a new domain or to multi-hop evidence spread across many chunks, the selected top-k will omit the answer.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic chunking keeps QA accurate as contexts reach 256k","Chunk at semantic gaps, select by question, read far longer","Question-aware dynamic chunking outperforms fixed slicing","Semantic chunking and selection for ultra-long context QA","Dynamic chunking at semantic boundaries boosts long-context QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00082,"raw_usage":{"total_tokens":3564,"prompt_tokens":891,"completion_tokens":2673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":2591}},"tokens_in":507,"tokens_out":2673,"duration_ms":17995,"temperature":1.0,"reasoning_tokens":2591,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:58:11.413755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier: on a held-out domain such as legal or medical documents, mark the single chunk that contains the gold answer, let the classifier rank all chunks, and compute whether the evidence chunk appears in the selected top-k. If the selection recall is no better than random for multi-hop questions whose evidence is spread across four or more chunks, the paper's robustness claim is overturned.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the semantic-splitting idea that low embedding similarity marks natural boundaries; DCS builds its chunker on it."}],"review_version":1}