{"id":"66b8273c-159e-4a9b-a641-39f9797369dc","arxiv_id":"2505.07793","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"OPRM, a training-free chunk-and-select inference method, improves recurrent LLMs by 14-51% on LongBench and sets a 7B-class LongBench v2 record, while raising doubts about whether recurrent models exploit long-range dependencies.","lead":"A new inference-time method that splits long contexts into chunks, processes each chunk separately, and decodes only from the most confident chunk dramatically improves recurrent LLMs on long-context benchmarks. The results also suggest these models rarely use information spread across distant parts of the input.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OPRM's central claim that min-entropy selects the most relevant chunk is never directly validated; aggregate gains may be driven by overflow avoidance alone, since random chunking already beats full context and max-query-probability selection is near-random.","rationale":"The reader's weakest assumption is exactly that the model's own output confidence (minimum entropy) reliably identifies the relevant chunk, and that the information needed is localized to a single chunk. My stress-test agrees with this as the most load-bearing assumption. The paper's controlled AR experiments provide credible evidence for a fixed memory-capacity bottleneck, and the fact that random chunk selection outperforms full-context processing at long lengths is independent support for overflow avoidance. However, the method's distinctive claim is not merely that chunking helps, but that OPRM 'identifies and processes only the most relevant portion' and that this single-chunk strategy outperforms full-context processing even on cross-context tasks. That claim depends on the reliability of the min-entropy selector. The paper provides only aggregate benchmark improvements as evidence for the selector, and its own Table 5 ablation shows that confidence-based selection is only modestly better than random in some length regimes, while max-query-probability selection is sometimes worse than random. No ground-truth selection accuracy is reported, so the causal story remains unvalidated. This is a concrete, checkable gap rather than an internal inconsistency, and it is consistent with the paper's own stated limitation in Appendix C that single-chunk decoding cannot capture all global dependencies. Since the reader already returned CONDITIONAL and this concern reinforces rather than overturns that verdict, I recommend no change to the reader's verdict.","tokens_in":31894,"tokens_out":7178,"duration_ms":82652,"concrete_test":"Run OPRM with the reported settings (L=3000, IDK filter where applicable) on a subset of LongBench-e multi-document QA tasks (HotpotQA and 2WikiMQA) where gold supporting paragraphs are available. Map each gold evidence paragraph to the OPRM chunk that contains it, and compute: (1) top-1 selection accuracy of min-entropy selection versus random selection; (2) end-task F1 when decoding from the oracle gold chunk versus the min-entropy chunk and versus a random chunk, with the same IDK filter and decoding settings. If min-entropy top-1 accuracy is not significantly above random, or if oracle-chunk F1 substantially exceeds min-entropy F1, then the 'most relevant chunk' mechanism is unsupported and the reported gains are primarily overflow avoidance. If min-entropy accuracy approaches the oracle ceiling, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism of OPRM, as stated in Algorithm 1 and Appendix C, is that the model's output distribution after seeing each chunk identifies the chunk containing the answer-relevant information, and decoding from that single chunk is what drives the gains. The paper never directly measures whether min-entropy selection actually selects the chunk containing the gold evidence, nor does it report selection accuracy, calibration statistics, or error bars for the selection step. This matters because the paper's own ablation in Table 5 shows that random chunk selection already outperforms the full-context baseline on Falcon-Mamba-Inst-7B at 4K-8K and 8K+ (23.02 vs 21.18 and 27.62 vs 18.4), and the max-Pr(Q|[P,C_i]) criterion barely beats random at 4K-8K (26.13 vs 23.02) and is worse at 8K+ (25.76 vs 27.62). Thus a substantial part of OPRM's improvement may be explained by overflow avoidance and context shortening alone, not by reliably identifying the most relevant chunk. Without a direct test of the selection mechanism, the claim that the method 'processes only the most relevant portion' and the stronger inference that recurrent models do not exploit cross-chunk dependencies are not established; the aggregate benchmark gains are consistent with the weaker statement that any chunk is better than an overflowing full context.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how fixed-size recurrent memory limits long-context performance in recurrent LLMs. It introduces an associative recall (AR) diagnostic showing that accuracy degrades as the number of key-value facts grows, even when the sequence is padded to fixed length, and it reproduces this overflow-like pattern in small 2-layer Mamba models trained from scratch across different channel and state dimensions. The paper then proposes OPRM, a training-free inference method that splits the context into chunks, processes each chunk with the prefix and suffix in parallel, removes chunks predicting an IDK/error token, selects the chunk with lowest output entropy, and decodes the answer from that chunk's recurrent state. On LongBench the method improves average scores by 14-51% over full-context baselines depending on the model, substantially improves multi-hop QA, extends Mamba-1.4b to longer contexts than dedicated context-extension methods, and achieves a LongBench v2 score of 30.8 with Falcon3-Mamba-Inst-7B, which the paper identifies as state of the art for the 7B size class. The manuscript includes ablations of chunk selection and the IDK filter, comparisons to RAG and agentic baselines, and an efficiency analysis.","tokens_in":32107,"tokens_out":7879,"duration_ms":70936,"significance":"The overflow phenomenon is convincingly demonstrated in the controlled setting: AR accuracy decays with the number of facts, is largely insensitive to padded sequence length, and improves with state size without disappearing. This is a useful diagnostic and the paper's main empirical contribution. OPRM is simple, training-free, parallelizable, and clearly better than full-context decoding on most tested benchmarks, so the method itself is valuable even if the selection mechanism is not fully understood. The paper releases code and detailed experimental configurations, which strengthens reproducibility. However, the most provocative interpretation, that recurrent LLMs do not exploit cross-chunk dependencies, is not established by the current experiments, and the headline gains depend on per-benchmark selection of the chunk size. With additional validation of the selection mechanism and a stricter evaluation protocol, this would be a solid contribution to the long-context efficiency literature.","major_comments":[{"comment":"The central claim that OPRM 'identifies and processes only the most relevant portion' is not directly validated. No experiment measures whether the min-entropy criterion selects the chunk containing the gold evidence; the evidence is indirect, consisting of final task accuracy plus one qualitative example in Figure 7. This matters because Table 5 shows that random chunk selection already outperforms full-context decoding in the 4K-8K and 8K+ groups (23.02 vs 21.18 and 27.62 vs 18.4), so a large part of the gain is attributable to overflow avoidance rather than to localization. Min entropy is stronger than random in those groups (37.1 and 35.18), which shows the selection signal has value, but the aggregate numbers remain consistent with the weaker hypothesis that any short chunk is better than an overflowing full context. Please report selection accuracy against a gold-chunk oracle (e.g., the chunk containing the supporting evidence in HotPotQA or MuSiQue), entropy calibration statistics, and variance of the selection step; this is needed to support the localization claim and the related claim about cross-chunk dependencies.","section":"§4.2, Algorithm 1, Table 5"},{"comment":"The headline improvements are computed with per-benchmark oracle selection of the chunk size: Appendix A.3 states that L∈{1000,2000,3000} is used and 'we select the best scoring chunk size.' Since the abstract and Figure 1 (right) quote improvements of 14-51%, these numbers are upper bounds over L, not the performance of a fixed configuration. Table 7 shows that scores are fairly robust across L for many LongBench tasks, but the selection is still made on the test set. Please report results for a single fixed L, or for a validation-based selection rule, for all headline aggregates, and give the spread of the aggregate scores across L rather than only the per-task sigma/mu values.","section":"Appendix A.3 and Figure 1 (right)"},{"comment":"The abstract and Section 7 say the results 'raise questions about whether recurrent models genuinely exploit long-range dependencies,' but the evidence does not discriminate between 'recurrent models cannot use cross-chunk information' and 'full-context decoding is impaired by overflow, so any chunking helps.' The paper's own multi-chunk ablation (Table 11) shows that combining the top-k chunks (CC) outperforms OPRM at 0-4K (40.27 vs 37.41) and is comparable at 8K+ (35.98 vs 36.25), with a higher overall average (37.27 vs 36.05), and no task is annotated for whether it actually requires cross-chunk evidence. Please either soften the conclusion to an explicitly open question supported by a task-level analysis, or add a controlled experiment in which the answer requires evidence from two different chunks and compare OPRM, CC, and full-context decoding under matched token budgets.","section":"§1/§7 and Appendix B.4, Table 11"}],"minor_comments":[{"comment":"The zero-shot AR curve is averaged over only 5 sampled contexts per number of facts; please add error bars or report additional seeds, since this curve is the primary motivation for the method.","section":"Section 3 and Appendix A.1"},{"comment":"The experimental configuration is heterogeneous across models: the IDK filter is applied to Falcon-Mamba-Inst-7B, RecurrentGemma-IT-9B, and RWKV6-Finch-7B but not to Falcon3-Mamba-Inst-7B, and summarization tasks use a different decoding mode; this should be stated in the main text so the gains are not read as a single uniform intervention.","section":"Appendix A.3 and Figure 1 (right)"},{"comment":"The 'new SOTA for this size class' claim is made on the full LongBench v2 dataset, while Table 1 compares to baselines only on the <150K-token subset; please clarify that the full-dataset improvement over the unmodified Falcon3-Mamba-Inst-7B is not directly measured because the baseline runs out of memory, and that the comparison to Transformer models therefore uses different evaluation subsets across the two tables.","section":"Tables 1 and 2"},{"comment":"The sentence 'OPRM outperforms vanilla inference in speed' is true only for contexts of 8K and above in Table 8; at 2K the OPRM time is 2.2s vs 2.0s and at 4K the times are equal, so the speed claim should be qualified as applying to longer contexts.","section":"Section 5.4, Table 8"},{"comment":"There is a typo in 'During evaluation the same setting is usedm' which should read 'used'; please fix it.","section":"Appendix A.5"}],"recommendation":"major_revision","confidential_remarks":"This is a promising empirical paper. The main revision should focus on validating the selection mechanism and reporting results under a fixed protocol; both are achievable without changing the method. The interpretive claims about cross-chunk dependencies must be softened or supported by additional experiments. The paper fits the scope of COLM, and the code release and controlled AR experiments are strengths."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper documents a real phenomenon—recurrent LLMs overflow on associative recall even with short contexts—and shows that a simple chunk-and-select inference method gives large, consistent gains on long-context benchmarks. The central finding is solid; the weaker parts are the un-validated selection mechanism and some post-hoc benchmark choices. \n\nWhat's new and good: the AR curves in Fig. 1 and the controlled training in Fig. 2 cleanly separate information load from context length, showing capacity is the bottleneck, not length generalization. OPRM is simple, training-free, and the gains are not subtle: 14–51% on LongBench and a 30.8 LongBench v2 score for Falcon3-Mamba. The paper includes code, detailed implementation notes, and honest limitations about cross-chunk fusion. That is real, reproducible work. \n\nThe soft spots, in proportion: the stress-test is half right. Random chunking beating full-context on long inputs (Table 5) shows overflow avoidance alone explains part of the gain. But min-entropy clearly beats random at 4–8K and 8K+ (37.1 vs 23.0; 35.2 vs 27.6), so the selection is doing real work at longer contexts, not just the chunking. What is missing is a direct check: does min-entropy pick the chunk containing the gold evidence? The paper never measures selection accuracy or calibration, and Appendix C states the assumption rather than testing it. So the claim that OPRM 'processes only the most relevant portion' is not fully established, and the broader claim that recurrent models don't exploit long-range dependencies is a reasonable inference but not proven. \n\nThe benchmark numbers also come with more adaptation than the abstract suggests: chunk size is selected on the test set (best of three), the IDK filter is toggled per model, and summarization uses a separate decode-all-chunks variant. That inflates the headline gains to some degree. The below-chance RWKV result on LongBench v2 is odd and unexplained, and the main tables lack error bars. \n\nBottom line: the central finding—recurrent LLMs underuse long contexts because of fixed memory capacity—is well supported, and OPRM is a practical method worth knowing. I'd send it to review, with requests for selection validation and a cleaner evaluation protocol. It also makes for a good reading-group discussion on where inference-time heuristics can and cannot carry a claim.","headline":"The overflow finding is real and OPRM helps consistently; the selection mechanism is under-validated and the benchmark gains include some test-set tuning, but the core result holds.","tokens_in":32710,"tokens_out":3667,"would_cite":true,"duration_ms":36117,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Recurrent LLMs trained on long contexts still overflow their fixed memory; a chunk-based inference method that decodes only the most relevant chunk fixes the failure and beats full-context processing.","keywords":["recurrent LLMs","memory overflow","associative recall","chunk-based inference","long-context reasoning","selective decoding","LongBench","context extension"],"falsifier":"Build a benchmark where every question can be answered only by combining facts from two chunks placed far apart, so that no single chunk contains the full answer; if OPRM scores lower than full-context decoding on those questions while matching it on single-chunk questions, the locality assumption is the ceiling.","tokens_in":31651,"feed_emoji":"🧩","tokens_out":6726,"duration_ms":62697,"temperature":0.7,"pith_summary":"Recurrent language models keep a fixed-size memory as they read, and this paper argues that even state-of-the-art models trained on long contexts do not actually use those contexts: the memory overflows, so retrieval accuracy collapses as the number of facts grows. The paper proposes OPRM, a training-free inference procedure that splits the prompt into chunks, processes each chunk in parallel, and decodes only from the chunk whose output distribution is most confident. In experiments across LongBench, LongBench v2, and context-extension tasks, this single-chunk strategy often beats full-context processing, improving scores by 14% to 51% on LongBench and reaching a reported state-of-the-art 30.8 on LongBench v2 for Falcon3-Mamba-Inst-7B. The result matters because it suggests recurrent LLMs' long-context weakness is a memory-capacity problem, not a length-generalization problem, and it can be mitigated without retraining.","feed_headline":"Chunked inference boosts recurrent long-context LLMs by up to 51 percent","feed_subtitle":"Processing only the most relevant chunk beats full-context decoding, even on tasks expected to need cross-context reasoning.","key_machinery":"The load-bearing object is the fixed-size recurrent hidden state that recurrent layers update token by token; its capacity limits how many facts can be retrieved. In a Mamba layer this state has dimension $d \\times d_{\\mathrm{state}}$. OPRM (Overflow Prevention for Recurrent Models) is the method built around that limitation: it splits the context $C$ into $b$ chunks of length $L$, runs each chunk as a separate prompt $[P, C_i, S]$ in parallel, filters out chunks whose first predicted token is an \"Error\" IDK token, then selects the chunk with the lowest entropy over the next-token distribution and decodes from its recurrent state. This avoids overflow because each chunk's information load is bounded, and the associative-recall curve serves as the diagnostic that reveals the overflow in the first place.","core_discovery":"The central discovery is a bounded-memory overflow failure in recurrent LLMs: when a prompt contains more key-value facts than the fixed recurrent state can hold, retrieval accuracy falls toward zero, and the amount of information matters more than sequence length. The paper establishes this with associative-recall curves on Falcon-Mamba-Inst-7B and with trained 2-layer Mamba models, and then shows that OPRM, which turns the prompt into chunks small enough to fit memory and selects the most relevant one by minimum entropy after discarding \"don't know\" chunks, prevents the overflow. The same method that avoids overflow often improves over full-context decoding even on tasks that seem to require cross-context relations, and it extends usable context lengths well beyond training. The paper states this as evidence that recurrent models underuse their long contexts and that single-chunk decoding can be state-of-the-art.","pith_inferences":["If recurrent models underuse trained long contexts because their memory overflows, then the advertised \"context length\" is not the operative capacity; the useful context is bounded by memory capacity, and evaluations should report information density per input rather than token count alone.","OPRM's success suggests that a recurrent LLM's own next-token confidence can act as a cheap relevance oracle, a signal that could be reused for retrieval-augmented generation or for routing long prompts without an external retriever.","A natural extension the paper does not build is to select multiple chunks and merge their states with an overflow-aware aggregation, since its multi-chunk ablation shows gains on short contexts; such a hybrid could recover tasks that genuinely need cross-chunk evidence.","The method predicts that OPRM's advantage grows with context length and information density, so applying it to larger recurrent or hybrid models should show smaller overflow but still monotone gains until memory scales with the task."],"forward_implications":["On LongBench, OPRM raises overall scores by 14% for Falcon3-Mamba-Inst-7B, 28% for Falcon-Mamba-Inst-7B, 50% for RecurrentGemma-IT-9B, and 51% for RWKV6-Finch-7B, with the largest gains appearing on the longest contexts.","On LongBench v2, Falcon3-Mamba-Inst-7B plus OPRM reaches 30.8, the reported state of the art at its size class, and it beats comparable 7B Transformers on the 32K-128K and 128K+ length groups.","OPRM extends contexts far beyond training length without fine-tuning: a Mamba-130m trained on 2K tokens keeps high needle-in-a-haystack accuracy at 512K tokens, 256 times its training length, versus 64 times for a dedicated extension method.","Multi-hop reasoning scores double or more on several LongBench benchmarks, so overflow prevention helps tasks that require chaining evidence, not just single-fact retrieval.","Because chunks are processed in parallel and each stored state is tiny, OPRM is faster than full-context decoding on long inputs, for example 26.9 seconds versus 36.2 seconds at 128K tokens for Falcon3-Mamba-Inst-7B, with a small memory overhead."],"supporting_citations":[{"why":"Supplies the associative recall task and Zoology-style measurement used to reveal the overflow curves.","marker":"Arora et al. (2023)"},{"why":"Provides Falcon-Mamba-Inst-7B, the primary recurrent model whose zero-shot AR curve motivates OPRM.","marker":"Zuo et al., 2024"},{"why":"Provides LongBench, the main real-world benchmark on which OPRM reports 14-51% improvements.","marker":"Bai et al., 2024"},{"why":"Provides LongBench v2, the benchmark where OPRM reports a 30.8 state-of-the-art score at the 7B size class.","marker":"Bai et al., 2025"},{"why":"Establishes the fixed-memory-capacity scaling behavior that OPRM's chunking exploits.","marker":"Arora et al. (2024)"},{"why":"Supplies the DeciMamba baseline and the needle-in-a-haystack and document-retrieval setups used for context-extension comparisons.","marker":"Ben-Kish et al., 2025"},{"why":"Provides the LongMamba context-extension baseline beaten on LongBench e.","marker":"Ye et al., 2025"},{"why":"Provides Falcon3-Mamba-Inst-7B, the model that reaches the LongBench v2 state-of-the-art result under OPRM.","marker":"Team, 2024"}],"fun_headline_variants":["Chunked decoding fixes overflow, lifts recurrent LLMs up to 51%","Single-chunk inference outperforms full context in recurrent LLMs","Overflow fix: relevant chunk only boosts long-context LLMs","Recurrent LLMs underuse context? Chunk selection says yes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"OPRM assumes that everything needed to answer a question sits inside one fixed-size piece of the context, and that the model's least-uncertain prediction identifies that piece; tasks that require joining evidence from several distant pieces would break it.","fun_headline_variants_meta":{"raw":{"variants":["Chunked decoding fixes overflow, lifts recurrent LLMs up to 51%","Single-chunk inference outperforms full context in recurrent LLMs","Overflow fix: relevant chunk only boosts long-context LLMs","Recurrent LLMs underuse context? Chunk selection says yes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001372,"raw_usage":{"total_tokens":5554,"prompt_tokens":932,"completion_tokens":4622,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":4545}},"tokens_in":548,"tokens_out":4622,"duration_ms":31641,"temperature":1.0,"reasoning_tokens":4545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:07:26.641463+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a benchmark where every question can be answered only by combining facts from two chunks placed far apart, so that no single chunk contains the full answer; if OPRM scores lower than full-context decoding on those questions while matching it on single-chunk questions, the locality assumption is the ceiling.","supporting_citations":[],"review_version":1}