{"id":"70e46787-6d4a-4748-b472-f66af7eadbc4","arxiv_id":"2412.12486","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ACRE uses a bi-layer KV cache with query-guided refilling to answer long-context questions beyond a model's native window, reporting gains over RAG and compression baselines.","lead":"This paper introduces ACRE, a method that compresses long documents into two levels of key-value memory and refills the compact level with query-relevant details when answering questions. The approach is designed to let language models handle contexts far beyond their native window with less memory and compute, an important practical problem for AI systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage-2 training cannot improve the query-guided selection as described: the hard top-k in Eq. 16 has no gradient path, and Eq. 21 conditions on the full L2 cache, so the attention scores S may be an untrained heuristic rather than a learned refilling signal.","rationale":"The reader identified the L1-proxy fidelity and the query-to-L1 ranking as the weakest assumption. My stress test sharpens this into a specific, textually grounded gap: even if the L1 tokens were faithful proxies, the paper does not describe a training signal that could optimize the ranking S. Stage 1 optimizes a language-modeling loss that is blind to QA relevance, and stage 2's loss, as written, cannot push gradients through the hard top-k selection. This is an internal-consistency concern rather than a disagreement with external consensus, and it is load-bearing because the method's name and contribution are precisely the query-guided refilling mechanism. The concern is not fatal if a differentiable surrogate exists, but the manuscript must state it and the experiments must isolate its contribution. The existing ablations (Figure 3) control for SFT data but do not control for the selection policy, so they cannot distinguish a learned selector from a fixed heuristic. My recommended verdict remains CONDITIONAL, matching the reader's judgement; I therefore mark it UNCHANGED. I also credit the paper for its ablation of training stages and parameter-choice analysis, which would help resolve the concern if extended to random and oracle selection.","tokens_in":14630,"tokens_out":7290,"duration_ms":69702,"concrete_test":"Implement ACRE as specified, but after stage 1 freeze every parameter that affects S (the L1 key projections and L1 input projections) and run stage 2 using exactly the loss in Eq. 21. At inference, compare three selection policies with the same k: (a) ACRE's query-based S, (b) uniform-random block selection, and (c) oracle selection using gold-answer block overlap. Measure task accuracy and the overlap between selected blocks and oracle-relevant blocks on LongBench, InfiniteBench, and UltraDomain subsets. If (a) and (b) are statistically indistinguishable and (c) is higher, the query-guided selection is not the source of gains. Alternatively, if the authors claim a differentiable training path, provide the explicit surrogate used to approximate Eq. 16 and verify its gradient with a finite-difference check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on query-guided activation refilling (Section 2.4, Eqs. 15-17): the query attends to L1 keys, the pooled scores S choose L2 blocks, and those blocks are inserted into the L1 cache for decoding. For this to be a learned mechanism, some training signal must improve S. Stage 1 (Eq. 20) is a language-modeling objective over L1 and L2 tokens; it trains L1 representations to predict future text, not to rank blocks by answer relevance. Stage 2 (Eq. 21) is a next-answer loss, but the selection in Eq. 16 is a discrete arg topk. The loss depends on the selected set I only through the resulting KV cache, and the gradient of that dependence with respect to the parameters producing S is zero almost everywhere because a small change in S does not change I unless a score is exactly at the top-k boundary. Hence Eq. 21 cannot update the ranking, unless the paper uses a differentiable surrogate (e.g., Gumbel top-k, soft attention, or REINFORCE), which the manuscript does not describe. Additionally, the notation in Eq. 21, P(yt | XL2, q), appears to condition on the full L2 context rather than the selected subset, making it unclear how gradients would flow through the refilling operation at all. If the query-to-L1 scores are not trained for QA relevance, the reported gains may come from the L1 cache plus task-specific fine-tuning, with the refilled blocks contributing no better than random block insertion; the mechanism's claimed advantage over RAG and compression baselines would then be unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ACRE (ACtivation REfilling), a method for long-context information-seeking tasks. ACRE interleaves trainable L1 tokens into the input context to form a compact L1 KV cache that proxies local blocks of L2 tokens, retained as a detailed L2 cache that can be offloaded to CPU memory. During decoding, the query attends to the L1 cache and the resulting attention scores are used to select and refill the L1 cache with corresponding blocks from the L2 cache. Training is done in two stages: stage 1 learns the L1 proxy representations through a next-token objective on long text, and stage 2 fine-tunes on QA data. Experiments on 12 long-context QA datasets with Qwen2.5-3B, Qwen2.5-7B, and Llama3.2-3B report consistent gains over full-context, compression, sparse-attention, and RAG baselines, together with efficiency measurements at up to 1024K tokens.","tokens_in":15001,"tokens_out":3601,"duration_ms":33141,"significance":"If the method works as claimed, it addresses a practically important problem: answering queries over contexts far longer than an LLM's native window while using less GPU memory than full-context processing. The paper's empirical scope is broad (12 datasets, 3 backbones, long-context efficiency measurements), and the ablations test the contribution of each training stage and the effect of the L1/L2 interval and refilling length. The central mechanism, however, has an unresolved training-signal question: the hard top-k selection in Eq. (16) has no gradient path, and the stage-2 loss in Eq. (21) does not, as written, depend on the selected set. In addition, training data and hyperparameter selection overlap with the evaluation benchmarks, and no code or error bars are provided, so the empirical claims cannot yet be fully verified. These issues are load-bearing; the recommendation is major revision.","major_comments":[{"comment":"The paper does not establish that stage-2 training can improve the query-guided selector. The selection I = arg topk(S) in Eq. (16) is a discrete operation, so the gradient of any loss with respect to the parameters producing S is zero almost everywhere unless a differentiable surrogate (e.g., Gumbel top-k, soft attention, or REINFORCE) is used; no such surrogate is described. Furthermore, Eq. (21) defines the stage-2 loss as −Σ log P(y_t | X_L2, q), which conditions on the full L2 cache rather than on the refilled cache determined by I, so it is unclear whether the loss depends on the selection at all. Since 'query-guided activation refilling' is the paper's central contribution, the authors should either specify an explicit training signal for the selector and correct Eq. (21), or present evidence that the attention scores S are already trained for QA relevance by stage 1.","section":"§2.4–2.5, Eqs. (16)–(21)"},{"comment":"There is a risk of benchmark contamination in the reported gains. Stage-2 training uses synthetic data from Qian et al. (2025b), which is the same source as the UltraDomain Complex QA evaluation sets, and from Zhang et al. (2024a); the evaluation includes those benchmark families. The paper should clarify the exact overlap between training and evaluation examples, and ideally report results on held-out benchmarks or with leave-one-benchmark-out training. Relatedly, the main hyperparameters (l = 16, η = 4096, W = 32K) are stated in Appendix A without a separate validation procedure, and Figure 4 examines these parameters on evaluation datasets (nar, en.qa, cs), suggesting selection on test data. The authors should specify how hyperparameters were chosen and whether any evaluation set was used for model selection.","section":"§3.1, §3.3, Appendix A"},{"comment":"The empirical comparison lacks uncertainty estimates. Table 1 reports single numbers with no variance across seeds, and several differences are small (e.g., 2WikiMQA: 36.4 vs 36.3; MuSiQue: 26.2 vs 26.0), so it is not possible to judge whether ACRE's gains are significant on those datasets. Table 2 similarly reports averages over 20 samples without variance or a detailed measurement protocol (e.g., whether CPU-offload transfer time for the L2 cache is included in latency). Reporting means over multiple runs with standard deviations, or at least specifying the number of seeds and the significance of differences, would substantially strengthen the empirical claims.","section":"Table 1 and Table 2"},{"comment":"The paper does not state how positional encodings are handled when L1 tokens are interleaved. With RoPE-based models, inserting an L1 token at position l shifts all subsequent L2 tokens to new positional indices, which can degrade the pretrained model's longer-range attention unless compensated. The manuscript neither describes such compensation nor verifies empirically that positional shifts are harmless; this is a load-bearing assumption for the bi-layer cache construction and should be addressed explicitly.","section":"§2.3, Eq. (5), and positional encodings"}],"minor_comments":[{"comment":"The displayed nested sequence in Eq. (5) is incomplete: it shows only one L1 token after x_l and one after x_n, rather than one L1 token after every l L2 tokens. The same issue appears in Eqs. (7) and (8). The intended pattern should be written, e.g., (x_1,...,x_l, x^{L1}_1, x_{l+1},...,x_{2l}, x^{L1}_2, ...).","section":"§2.3, Eq. (5)"},{"comment":"There are several presentation issues: 'a information-seeking task' should be 'an information-seeking task'; the reference '(Zhang et al., 2024b; ?)' contains a missing citation; and the phrase 'In summary, ACRE outperforms directly using vanilla LLMs' is grammatically awkward and should be rephrased.","section":"§1 and §2.1"},{"comment":"The limitation statement notes that ACRE adds about 17.2% parameters, but Table 2 shows that at 64K context ACRE uses more GPU memory than the vanilla model (20.8 vs 18.5 GiB for Qwen2.5-3B). The paper should reconcile this with the claim that ACRE reduces overall GPU memory, e.g., by clarifying that the benefit appears only at longer context lengths.","section":"Limitation section"},{"comment":"The description of stage-2 training contains an inconsistency: 'The model is trained for three epochs with a batch size of 8 and a learning rate of 1 × 10−5 for two epochs.' This should be corrected to specify the learning rate for each of the three epochs.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising direction and a broad evaluation, but the reviewer's central concern about the absence of a gradient path for the refilling selector is serious and should be resolved before publication. The lack of code, absence of error bars, and potential training/evaluation overlap also make the empirical claims difficult to verify. The manuscript may be suitable after major revision if the authors can clarify the training objective and provide stronger evidence that the query-guided selection is actually learned."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's main contribution is the bi-layer KV cache: interleaving trainable L1 proxy tokens into the context, computing keys/values with a small working window, and then using query attention over L1 to refill from L2. That structure is new relative to the cited work and is a reasonable way to extend effective context beyond the native window while bounding GPU memory. The experimental coverage is broad—12 datasets, 3 backbones—and the efficiency numbers look plausible. The soft spot is the central claim that the refilling selection is learned. The stress test is right: stage-2 training as described in Eq. 21 conditions on the full L2 context, not on the selected subset, and the top-k in Eq. 16 has no gradient path to the attention scores S. Nothing in the text explains a differentiable surrogate, so the query-guided scores appear to be a heuristic derived from the L1 projections, not a trained ranking. The ablation 'w/o Refilling' shows that the heuristic helps, but that is not the same as learning to retrieve. This is a real gap, not a nitpick, and it undermines the paper's stated contribution. Other issues are more standard: hyperparameters (l, eta, W) appear to be chosen on the evaluation tasks, there are no error bars anywhere, and the code is not released despite the abstract promising it. Stage-2 training also uses synthetic data from the same lab's UltraDomain/MemoRAG family, which overlaps with a major part of the evaluation. These reduce confidence but are not fatal. The paper is worth a serious referee. The framework is plausible and the engineering is careful; the fix is to clarify the training-time forward pass—whether selection is soft or REINFORCE-like—or to honestly reframe the refilling as a fixed non-learned heuristic. Without that, the reported gains could be mostly from the L1 cache plus task fine-tuning. I would recommend sending it to peer review, but the authors must address the gradient issue explicitly before publication.","headline":"The bi-layer cache is a solid engineering idea, but the query-guided refilling mechanism is not actually trained as described—a fixable flaw, but one the paper must address.","tokens_in":586,"tokens_out":650,"would_cite":false,"duration_ms":34252,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-tier key-value cache with query-guided refilling lets LLMs answer questions over contexts far beyond their native window while using less memory than full-context processing.","keywords":["long-context LLM","KV cache compression","query-guided refilling","bi-layer cache","information-seeking QA","selective attention","retrieval-augmented generation","context window extension"],"falsifier":"Give ACRE a long-context query whose correct answer requires evidence spread across many blocks while its surface wording matches only a few topic-similar distracting blocks, such as a compare-all-experiments question over a long scientific article. If ACRE's top-k refilling selects only the blocks the query attends to and answer quality drops below a full-context baseline, the query-guided selection and L1 proxying are not faithfully locating the evidence. A more direct check is to measure block-level reconstruction error between L1 tokens and their proxied L2 key-value activations on held-out text; large error would falsify the proxying assumption the whole method rests on.","tokens_in":14439,"feed_emoji":"🧠","tokens_out":5591,"duration_ms":48279,"temperature":0.7,"pith_summary":"ACRE is a method for letting an LLM answer questions over documents much longer than its native context window without paying the full cost of processing every token. The paper's claim is that for information-seeking tasks, a query's needs range from a single local detail to a synthesis of the whole document, so the right cache is two-tiered: a compact L1 cache holding global information and a detailed L2 cache holding local blocks, with each L1 token trained to stand for a block of L2 tokens. At query time the model attends only to the L1 cache, reads which L1 entries matter, and refills those entries with their corresponding L2 blocks before decoding the answer. On benchmark tasks this combination beats full-context decoding, KV compression, and retrieval-augmented baselines on most datasets, and in scaling tests it keeps producing readable answers at 1M-token contexts where vanilla models run out of memory or produce nonsense.","feed_headline":"Two-tier cache lets small LLMs handle million-token contexts","feed_subtitle":"A compact global cache plus query-guided local refilling beats RAG and full-context baselines at lower cost.","key_machinery":"The load-bearing object is the bi-layer KV cache. L1 tokens, newly introduced trainable tokens inserted every l original tokens, carry their own key and value projections and act as semantic proxies for the preceding block of L2 tokens; the proxying relationship is what lets a query attending to a small L1 cache index into the large L2 cache. The other components are the selective attention used during prefilling, which performs full attention only within a working window while still attending to distant L1 tokens, and the query-guided refilling step, which uses the pooled query-to-L1 attention scores S to pick top-k L2 blocks and splice them into the decoding cache. A two-stage optimization first trains the proxies by next-token prediction over long text, then trains the selection behavior on QA pairs, freezing the original model weights throughout.","core_discovery":"On its own terms, the paper establishes that long-context information-seeking can be decomposed into a cheap global pass and a query-guided local refinement. The model prefills a long context once, producing a nested key-value cache: L1 tokens interleaved among the original L2 tokens, where each L1 token is a learned semantic proxy for the preceding l L2 tokens, and a selective attention mechanism keeps the prefilling cost bounded by a working window. For a new query, the model computes attention from the query to the L1 cache, pools the scores across heads and query tokens, selects the top-k L1 positions, and swaps in the corresponding L2 key-value entries, producing a refilled cache that carries both global and query-specific local information. The two-stage training — next-token prediction to build faithful proxies, then QA fine-tuning to learn selection — updates only the newly introduced parameters. The experiments report consistent gains over the strongest available baselines, including full-context decoding of the same model, across 12 datasets, and the efficiency measurements show that the method handles 512K to 1M token inputs on a single 80 GB GPU where the vanilla model cannot.","pith_inferences":["A direct testable extension: ACRE's ceiling should depend on how compressible the context is. Documents with long stretches of independent facts should show a steeper accuracy drop as the L1/L2 interval grows, while highly local or structured texts should tolerate larger intervals better.","Because the L2 cache is offloaded to CPU memory, the refilling step becomes bandwidth-bound; the paper's efficiency numbers imply a prediction that latency grows with the number of selected blocks, which a follow-up could verify by profiling refill time against the refilling length.","The stage-1 training objective, predicting the next token from L1 plus recent tokens, rewards proxies that support local prediction rather than proxies that preserve every fact, so tasks requiring facts that are never predicted nearby are a plausible failure mode the benchmarks may not stress.","The same cache-and-refill recipe could apply to multi-turn dialogue over a fixed document set, where each turn refills a different subset of blocks; error accumulation across turns is a risk the paper does not address."],"forward_implications":["ACRE processes contexts far beyond the model's native window: at 512K and 1M tokens it fits on a single 80 GB GPU and produces readable answers where the vanilla model hits out-of-memory or nonsense.","Since the bi-layer cache is built once per context and reused across queries, the prefilling cost amortizes over many questions about the same document.","The refilling-length analysis implies a tunable precision-versus-efficiency knob: local-fact queries degrade if too many blocks are refilled, while global-integration queries keep improving with longer refills.","The L1/L2 interval trades compactness against fidelity: larger intervals shrink the L1 cache but overload each proxy token and reduce accuracy."],"supporting_citations":[{"why":"Supplies the LongBench single- and multi-document QA test sets used in the main evaluation.","marker":"[Bai et al., 2024]"},{"why":"Supplies the InfiniteBench En.QA and longer-context evaluation data, including contexts beyond 100K tokens.","marker":"[Zhang et al., 2024c]"},{"why":"Provides the UltraDomain datasets, the MemoRAG global-memory baseline ACRE must beat, and part of the stage-2 training data.","marker":"[Qian et al., 2025b]"},{"why":"The Beacon activation-compression baseline whose trainable token idea ACRE's L1 proxies extend, and a source of stage-2 synthetic QA data.","marker":"[Zhang et al., 2024a]"},{"why":"Supplies the RedPajama long-text corpus used for stage-1 training of the proxy tokens.","marker":"[Soboleva et al., 2023]"},{"why":"Supplies the LongAlpaca QA data used in stage-2 training to learn query-guided refilling.","marker":"[Chen et al., 2024]"},{"why":"Defines the underlying Qwen2 model family and its 128K native window, which sets the baseline for what counts as exceeding the context limit.","marker":"[Yang et al., 2024]"}],"fun_headline_variants":["Query-guided activation refilling: long-context QA with a bi-layer cache","Bi-layer KV cache: global overview, query-driven local refill","Compact global cache plus selective refilling tackles 1M-token inputs","ACRE: query-guided cache refilling for efficient long-context QA","Two-tier cache with query-pooled selection: 1M tokens on one GPU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes each L1 token is a faithful summary of the block of original tokens it stands for, and that the query's attention distribution over L1 tokens points to the blocks that actually contain the answer; if either fails, refilling pulls in the wrong text and ACRE degenerates into a lossy compressed cache.","fun_headline_variants_meta":{"raw":{"variants":["Query-guided activation refilling: long-context QA with a bi-layer cache","Bi-layer KV cache: global overview, query-driven local refill","Compact global cache plus selective refilling tackles 1M-token inputs","ACRE: query-guided cache refilling for efficient long-context QA","Two-tier cache with query-pooled selection: 1M tokens on one GPU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000414,"raw_usage":{"total_tokens":2162,"prompt_tokens":992,"completion_tokens":1170,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":1071}},"tokens_in":608,"tokens_out":1170,"duration_ms":9634,"temperature":1.0,"reasoning_tokens":1071,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:02:30.558025+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give ACRE a long-context query whose correct answer requires evidence spread across many blocks while its surface wording matches only a few topic-similar distracting blocks, such as a compare-all-experiments question over a long scientific article. If ACRE's top-k refilling selects only the blocks the query attends to and answer quality drops below a full-context baseline, the query-guided selection and L1 proxying are not faithfully locating the evidence. A more direct check is to measure block-level reconstruction error between L1 tokens and their proxied L2 key-value activations on held-out text; large error would falsify the proxying assumption the whole method rests on.","supporting_citations":[],"review_version":1}