Pith. sign in

REVIEW 4 major objections 4 minor 45 references

Boosting Long-Context Management via Query-Guided Activation Refilling

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-tier key-value cache with query-guided refilling lets LLMs answer questions over contexts far beyond their native window while using less memory than full-context processing.

desk verdict The bi-layer cache is a solid engineering idea, but the query-guided refilling mechanism is not actually trained as described—a fixable flaw, but one the paper must address. read the letter →

arxiv 2412.12486 v3 pith:4ACL3C6K submitted 2024-12-17 cs.CL cs.AIcs.IR

Hongjin Qian , Zheng Liu , Peitian Zhang , Zhicheng Dou , Defu Lian This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords long-contextLLMKVcachecompressionquery-guidedrefillingbi-layerinformation-seekingQAselectiveattentionretrieval-augmentedgenerationcontextwindowextension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ACRE is a method for letting an LLM answer questions over documents much longer than its native context window without paying the full cost of processing every token. The paper's claim is that for information-seeking tasks, a query's needs range from a single local detail to a synthesis of the whole document, so the right cache is two-tiered: a compact L1 cache holding global information and a detailed L2 cache holding local blocks, with each L1 token trained to stand for a block of L2 tokens. At query time the model attends only to the L1 cache, reads which L1 entries matter, and refills those entries with their corresponding L2 blocks before decoding the answer. On benchmark tasks this combination beats full-context decoding, KV compression, and retrieval-augmented baselines on most datasets, and in scaling tests it keeps producing readable answers at 1M-token contexts where vanilla models run out of memory or produce nonsense.

What carries the argument

The load-bearing object is the bi-layer KV cache. L1 tokens, newly introduced trainable tokens inserted every l original tokens, carry their own key and value projections and act as semantic proxies for the preceding block of L2 tokens; the proxying relationship is what lets a query attending to a small L1 cache index into the large L2 cache. The other components are the selective attention used during prefilling, which performs full attention only within a working window while still attending to distant L1 tokens, and the query-guided refilling step, which uses the pooled query-to-L1 attention scores S to pick top-k L2 blocks and splice them into the decoding cache. A two-stage optimization first trains the proxies by next-token prediction over long text, then trains the selection behavior on QA pairs, freezing the original model weights throughout.

What would settle it

Give ACRE a long-context query whose correct answer requires evidence spread across many blocks while its surface wording matches only a few topic-similar distracting blocks, such as a compare-all-experiments question over a long scientific article. If ACRE's top-k refilling selects only the blocks the query attends to and answer quality drops below a full-context baseline, the query-guided selection and L1 proxying are not faithfully locating the evidence. A more direct check is to measure block-level reconstruction error between L1 tokens and their proxied L2 key-value activations on held-out text; large error would falsify the proxying assumption the whole method rests on.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that long-context information-seeking can be decomposed into a cheap global pass and a query-guided local refinement. The model prefills a long context once, producing a nested key-value cache: L1 tokens interleaved among the original L2 tokens, where each L1 token is a learned semantic proxy for the preceding l L2 tokens, and a selective attention mechanism keeps the prefilling cost bounded by a working window. For a new query, the model computes attention from the query to the L1 cache, pools the scores across heads and query tokens, selects the top-k L1 positions, and swaps in the corresponding L2 key-value entries, producing a refilled cache that carries both global and query-specific local information. The two-stage training — next-token prediction to build faithful proxies, then QA fine-tuning to learn selection — updates only the newly introduced parameters. The experiments report consistent gains over the strongest available baselines, including full-context decoding of the same model, across 12 datasets, and the efficiency measurements show that the method handles 512K to 1M token inputs on a single 80 GB GPU where the vanilla model cannot.

Load-bearing premise

The method assumes each L1 token is a faithful summary of the block of original tokens it stands for, and that the query's attention distribution over L1 tokens points to the blocks that actually contain the answer; if either fails, refilling pulls in the wrong text and ACRE degenerates into a lossy compressed cache.

Editorial extensions

If this is right

  • ACRE processes contexts far beyond the model's native window: at 512K and 1M tokens it fits on a single 80 GB GPU and produces readable answers where the vanilla model hits out-of-memory or nonsense.
  • Since the bi-layer cache is built once per context and reused across queries, the prefilling cost amortizes over many questions about the same document.
  • The refilling-length analysis implies a tunable precision-versus-efficiency knob: local-fact queries degrade if too many blocks are refilled, while global-integration queries keep improving with longer refills.
  • The L1/L2 interval trades compactness against fidelity: larger intervals shrink the L1 cache but overload each proxy token and reduce accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension: ACRE's ceiling should depend on how compressible the context is. Documents with long stretches of independent facts should show a steeper accuracy drop as the L1/L2 interval grows, while highly local or structured texts should tolerate larger intervals better.
  • Because the L2 cache is offloaded to CPU memory, the refilling step becomes bandwidth-bound; the paper's efficiency numbers imply a prediction that latency grows with the number of selected blocks, which a follow-up could verify by profiling refill time against the refilling length.
  • The stage-1 training objective, predicting the next token from L1 plus recent tokens, rewards proxies that support local prediction rather than proxies that preserve every fact, so tasks requiring facts that are never predicted nearby are a plausible failure mode the benchmarks may not stress.
  • The same cache-and-refill recipe could apply to multi-turn dialogue over a fixed document set, where each turn refills a different subset of blocks; error accumulation across turns is a risk the paper does not address.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ACRE (ACtivation REfilling), a method for long-context information-seeking tasks. ACRE interleaves trainable L1 tokens into the input context to form a compact L1 KV cache that proxies local blocks of L2 tokens, retained as a detailed L2 cache that can be offloaded to CPU memory. During decoding, the query attends to the L1 cache and the resulting attention scores are used to select and refill the L1 cache with corresponding blocks from the L2 cache. Training is done in two stages: stage 1 learns the L1 proxy representations through a next-token objective on long text, and stage 2 fine-tunes on QA data. Experiments on 12 long-context QA datasets with Qwen2.5-3B, Qwen2.5-7B, and Llama3.2-3B report consistent gains over full-context, compression, sparse-attention, and RAG baselines, together with efficiency measurements at up to 1024K tokens.

Significance. If the method works as claimed, it addresses a practically important problem: answering queries over contexts far longer than an LLM's native window while using less GPU memory than full-context processing. The paper's empirical scope is broad (12 datasets, 3 backbones, long-context efficiency measurements), and the ablations test the contribution of each training stage and the effect of the L1/L2 interval and refilling length. The central mechanism, however, has an unresolved training-signal question: the hard top-k selection in Eq. (16) has no gradient path, and the stage-2 loss in Eq. (21) does not, as written, depend on the selected set. In addition, training data and hyperparameter selection overlap with the evaluation benchmarks, and no code or error bars are provided, so the empirical claims cannot yet be fully verified. These issues are load-bearing; the recommendation is major revision.

major comments (4)
  1. [§2.4–2.5, Eqs. (16)–(21)] The paper does not establish that stage-2 training can improve the query-guided selector. The selection I = arg topk(S) in Eq. (16) is a discrete operation, so the gradient of any loss with respect to the parameters producing S is zero almost everywhere unless a differentiable surrogate (e.g., Gumbel top-k, soft attention, or REINFORCE) is used; no such surrogate is described. Furthermore, Eq. (21) defines the stage-2 loss as −Σ log P(y_t | X_L2, q), which conditions on the full L2 cache rather than on the refilled cache determined by I, so it is unclear whether the loss depends on the selection at all. Since 'query-guided activation refilling' is the paper's central contribution, the authors should either specify an explicit training signal for the selector and correct Eq. (21), or present evidence that the attention scores S are already trained for QA relevance by stage 1.
  2. [§3.1, §3.3, Appendix A] There is a risk of benchmark contamination in the reported gains. Stage-2 training uses synthetic data from Qian et al. (2025b), which is the same source as the UltraDomain Complex QA evaluation sets, and from Zhang et al. (2024a); the evaluation includes those benchmark families. The paper should clarify the exact overlap between training and evaluation examples, and ideally report results on held-out benchmarks or with leave-one-benchmark-out training. Relatedly, the main hyperparameters (l = 16, η = 4096, W = 32K) are stated in Appendix A without a separate validation procedure, and Figure 4 examines these parameters on evaluation datasets (nar, en.qa, cs), suggesting selection on test data. The authors should specify how hyperparameters were chosen and whether any evaluation set was used for model selection.
  3. [Table 1 and Table 2] The empirical comparison lacks uncertainty estimates. Table 1 reports single numbers with no variance across seeds, and several differences are small (e.g., 2WikiMQA: 36.4 vs 36.3; MuSiQue: 26.2 vs 26.0), so it is not possible to judge whether ACRE's gains are significant on those datasets. Table 2 similarly reports averages over 20 samples without variance or a detailed measurement protocol (e.g., whether CPU-offload transfer time for the L2 cache is included in latency). Reporting means over multiple runs with standard deviations, or at least specifying the number of seeds and the significance of differences, would substantially strengthen the empirical claims.
  4. [§2.3, Eq. (5), and positional encodings] The paper does not state how positional encodings are handled when L1 tokens are interleaved. With RoPE-based models, inserting an L1 token at position l shifts all subsequent L2 tokens to new positional indices, which can degrade the pretrained model's longer-range attention unless compensated. The manuscript neither describes such compensation nor verifies empirically that positional shifts are harmless; this is a load-bearing assumption for the bi-layer cache construction and should be addressed explicitly.
minor comments (4)
  1. [§2.3, Eq. (5)] The displayed nested sequence in Eq. (5) is incomplete: it shows only one L1 token after x_l and one after x_n, rather than one L1 token after every l L2 tokens. The same issue appears in Eqs. (7) and (8). The intended pattern should be written, e.g., (x_1,...,x_l, x^{L1}_1, x_{l+1},...,x_{2l}, x^{L1}_2, ...).
  2. [§1 and §2.1] There are several presentation issues: 'a information-seeking task' should be 'an information-seeking task'; the reference '(Zhang et al., 2024b; ?)' contains a missing citation; and the phrase 'In summary, ACRE outperforms directly using vanilla LLMs' is grammatically awkward and should be rephrased.
  3. [Limitation section] The limitation statement notes that ACRE adds about 17.2% parameters, but Table 2 shows that at 64K context ACRE uses more GPU memory than the vanilla model (20.8 vs 18.5 GiB for Qwen2.5-3B). The paper should reconcile this with the claim that ACRE reduces overall GPU memory, e.g., by clarifying that the benefit appears only at longer context lengths.
  4. [Appendix A] The description of stage-2 training contains an inconsistency: 'The model is trained for three epochs with a batch size of 8 and a learning rate of 1 × 10−5 for two epochs.' This should be corrected to specify the learning rate for each of the three epochs.

Circularity Check

2 steps flagged · score 4.0 of 10

No equation-level circularity, but the empirical evidence is partly self-referential: stage-2 training uses the same first-author benchmark family as the evaluation, and hyperparameters are analyzed on test datasets before the main configuration is fixed.

  1. self citation load bearing [Appendix A (Implementation Details, Stage 2) and Section 3.1 (Datasets)]
    "In stage 2, we collect 28,400 QA SFT data points from LongAlpaca (Chen et al., 2024) and synthetic data from (Zhang et al., 2024a; Qian et al., 2025b). ... We evaluate ACRE and all baseline models across 12 information-seeking tasks from three public long-context benchmarks: LongBench (Bai et al., 2024), InfiniteBench (Zhang et al., 2024c), and UltraDomain (Qian et al., 2025b)."

    Qian et al. 2025b is the authors' prior MemoRAG paper and is the source of the UltraDomain benchmark family used for evaluation, and the same citation supplies synthetic stage-2 training data. Consequently the reported gains on Complex QA tasks (Financial, Legal, Physics, Biology, Math, CS) are at least partly fit to the same benchmark family on which ACRE is tested, rather than being fully independent predictions. This is an evidential self-referential loop rather than a definitional reduction: the method's equations do not encode the test answers, and LongBench/InfiniteBench plus external baselines provide independent content, so the central derivation is not forced.

  2. other [Section 3.4 (Ablation, Impact of Parameter Choice) and Appendix A]
    "For the main experiments, we configure ACRE with an L1/L2 interval l of 16, a maximum refilling length η of 4,096, and the maximum working context window W of 32K tokens. ... To investigate their impact, we conduct experiments with different values of η and l. Figure 4 presents the results of this analysis."

    The hyperparameter analysis in Figure 4 is performed on nar, en.qa, and cs, which are also test datasets in the main results table, and the values that achieve good numbers in that analysis (l=16, η=4096) are then fixed for the headline experiments. This is test-set model selection: the main configuration is chosen using the same benchmark instances whose scores are later reported, so the headline gains are partially a product of the evaluation set rather than a prediction. It is not a closed derivation, because ACRE's output is still computed by the model and external benchmarks are also reported, but it weakens the independence of the empirical evidence.

full rationale

No step in the paper makes the predicted answer equal to the method's inputs by construction. The L1 cache is trained with a next-token objective (Eq. 20) that does not include the QA answer, and the refilling operation (Eqs. 15-19) is an inference-time selection; the reported numbers are compared against external baselines. The two flagged items are evidential loops, not formal reductions: stage-2 training borrows synthetic data from the same first-author benchmark family used in evaluation, and the L1/L2 interval and refilling length are analyzed on test datasets before the main configuration is fixed. I also considered the reviewer's concern that Eq. 21 cannot train the discrete top-k selection in Eq. 16 because the hard arg top-k has no gradient path; that is a genuine correctness and mechanism gap, but it is not a circularity, because the paper does not define the selection scores as the answer or fit the answer to them explicitly. It should be reported as a separate technical criticism rather than counted as circularity.

Assumptions & free parameters 3 free parameters · 7 assumptions · 1 invented entities

The method rests on trainable L1 tokens and several domain assumptions about proxy quality and attention-based selection. There are no new physical entities, but the L1 token is a new model component whose reliability is only shown internally.

free parameters (3)
  • L1/L2 interval l = 16 in main experiments; sampled from {8, 16, 32, 64, 128} in stage-1 training
    Controls compression ratio and how much context each L1 token must summarize; tuned on evaluation benchmarks (Figure 4).
  • maximum refilling length eta = 4096 in main experiments
    Caps the amount of L2 detail refilled into the L1 cache; sensitivity is shown on test tasks in Figure 4.
  • working context window W = 32K tokens
    Used in selective attention and in the top-k refilling formula (Eq. 17); hand-selected as a design choice.
assumptions (7)
  • domain assumption Each L1 token acts as a semantic proxy for the preceding l L2 tokens.
    Introduced in Section 2.3 (Eq. 5) and trained via stage-1 next-token prediction; there is no independent evidence that this proxy is faithful.
  • domain assumption Query-to-L1 attention scores identify the L2 blocks most relevant to the answer.
    Section 2.4, Eq. 15-16 use mean-pooled attention as the selection signal; if attention does not correlate with relevance, refilling is misdirected.
  • domain assumption Selective attention with a short working window W preserves enough information to build reliable L1 and L2 activations.
    Section 2.3, Eq. 13-14 prune distant L2 tokens; this assumes distant tokens are not needed during cache construction.
  • domain assumption Stage-1 next-token loss maximizes the semantic volume of L1 activations.
    Section 2.5, Eq. 20 uses cross-entropy as the proxy-quality objective; there is no direct guarantee that this yields good summaries for arbitrary queries.
  • domain assumption Stage-2 QA loss teaches both answer generation and L2 retrieval.
    Section 2.5, Eq. 21 trains solely on the final answer; retrieval quality is only indirectly supervised.
  • ad hoc to paper Positional encodings remain valid for interleaved L1 tokens without modification.
    Section 2.3, Eq. 5 builds a nested sequence but does not specify how positions are assigned to L1 tokens or whether L2 positions shift.
  • standard math Standard softmax attention (Eq. 4) and the Transformer architecture.
    Section 2.1 defines the attention computation used throughout the paper.
invented entities (1)
  • L1 token (trainable proxy token)
    purpose: Compact global cache entries that each summarize a block of l L2 tokens and serve as the selection handle for refilling.
    No external falsifiable handle; utility is only demonstrated within the paper's benchmarks and ablations. Conceptually similar to landmark tokens in prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Boosting Long-Context Management via Query-Guided Activation Refilling." pith.science (2026). https://pith.science/paper/4ACL3C6K

@misc{pith2026241212486,
  author       = {Pith},
  title        = {Pith review of: Boosting Long-Context Management via Query-Guided Activation Refilling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ACL3C6K}},
  note         = {Machine review of arXiv:2412.12486}
}
read the original abstract

Processing long contexts poses a significant challenge for large language models (LLMs) due to their inherent context-window limitations and the computational burden of extensive key-value (KV) activations, which severely impact efficiency. For information-seeking tasks, full context perception is often unnecessary, as a query's information needs can dynamically range from localized details to a global perspective, depending on its complexity. However, existing methods struggle to adapt effectively to these dynamic information needs. In the paper, we propose a method for processing long-context information-seeking tasks via query-guided Activation Refilling (ACRE). ACRE constructs a Bi-layer KV Cache for long contexts, where the layer-1 (L1) cache compactly captures global information, and the layer-2 (L2) cache provides detailed and localized information. ACRE establishes a proxying relationship between the two caches, allowing the input query to attend to the L1 cache and dynamically refill it with relevant entries from the L2 cache. This mechanism integrates global understanding with query-specific local details, thus improving answer decoding. Experiments on a variety of long-context information-seeking datasets demonstrate ACRE's effectiveness, achieving improvements in both performance and efficiency.

Figures

Figures reproduced from arXiv: 2412.12486 by the authors.

Figure 1
Figure 1. Comparison of ACRE, standard RAG, and efficient long LLMs for information-seeking tasks. Stan￾dard RAG retrieves evidence without full-context per￾ception, and long LLMs struggle with contexts exceed￾ing their native window. ACRE overcomes these limi￾tations with a resource-efficient bi-layer KV cache and query-guided refilling, capturing both global and local information while enhancing performance. often involves … view at source ↗
Figure 2
Figure 2. Overview of ACRE. (a) ACRE constructs the Bi-layer KV cache from a long context. (b) For an input [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Ablation Study on Model Design Variations [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Analysis of the maximum refilling length [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 4 canonical work pages

  1. [1]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.172 Longbench: A bilingual, multitask benchmark for long context understanding . In Proceedings of the 62nd Annual Meeting of the Association for Com...

    doi:10.18653/v1/2024.acl-long.172 2024

  2. [2]

    Chi - Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. https://doi.org/10.48550/ARXIV.2404.00610 RQ-RAG: learning to refine queries for retrieval augmented generation . CoRR, abs/2404.00610

    arXiv doi:10.48550/arxiv.2404.00610 2024

  3. [3]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2023 a . https://arxiv.org/abs/2309.07597 Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation . Preprint, arXiv:2309.07597

    arXiv 2023

  4. [4]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023 b . Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595

    arXiv 2023

  5. [5]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. https://arxiv.org/abs/2309.12307 Longlora: Efficient fine-tuning of long-context large language models . Preprint, arXiv:2309.12307

    arXiv 2024

  6. [6]

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599--4610

    2021

  7. [7]

    Zican Dong, Tianyi Tang, Lunyi Li, and Wayne Xin Zhao. 2023. A survey on long text modeling with transformers. arXiv preprint arXiv:2302.14502

    arXiv 2023

  8. [8]

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. https://arxiv.org/abs/2402.10171 Data engineering for scaling language models to 128k context . Preprint, arXiv:2402.10171

    arXiv 2024

Show all 45 references
  1. [9]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2024. https://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey . Preprint, arXiv:2312.10997

    2024 arXiv

  2. [10]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.580 Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational Li...

    2020 doi

  3. [11]

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint...

    2024 arXiv

  4. [12]

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. https://arxiv.org/abs/2401.01325 Llm maybe longlm: Self-extend llm context window without tuning . Preprint, arXiv:2401.01325

    2024 arXiv

  5. [13]

    Tom \' a s Kocisk \' y , Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, G \' a bor Melis, and Edward Grefenstette. 2018. https://doi.org/10.1162/TACL\_A\_00023 The narrativeqa reading comprehension challenge . Trans. Assoc. Comput. Linguistics, 6:317--328

    2018 doi

  6. [14]

    Canny, and Ian Fischer

    Kuang - Huei Lee, Xinyun Chen, Hiroki Furuta, John F. Canny, and Ian Fischer. 2024. https://openreview.net/forum?id=OTmcsyEO5G A human-inspired reading agent with gist memory of very long contexts . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna...

    2024

  7. [15]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469

    2024 arXiv

  8. [16]

    Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. 2023. Scaling laws of rope-based extrapolation. In The Twelfth International Conference on Learning Representations

    2023

  9. [17]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In Forty-first International Conference on Machine Learning

    2024

  10. [18]

    MetaAI. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  11. [19]

    Amirkeivan Mohtashami and Martin Jaggi. 2024. Random-access infinite context length for transformers. Advances in Neural Information Processing Systems, 36

    2024

  12. [20]

    OpenAI. 2023. Gpt-4 technical report. https://cdn.openai.com/papers/gpt-4.pdf

    2023

  13. [21]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations

    2023

  14. [22]

    Hongjin Qian, Zheng Liu, Chao Gao, Yankai Wang, Defu Lian, and Zhicheng Dou. 2025 a . Hawkbench: Investigating resilience of rag methods on stratified information-seeking tasks. arXiv preprint arXiv:2502.13465

    2025

  15. [23]

    Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Defu Lian, Zhicheng Dou, and Tiejun Huang. 2025 b . Memorag: Boosting long context processing with global memory-enhanced retrieval augmentation. In Proceedings of the ACM on Web Conference 2025, pages 2366--2377

    2025

  16. [24]

    Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Yujia Zhou, Xu Chen, and Zhicheng Dou. 2024. https://arxiv.org/abs/2405.15318 Are long-llms a necessity for long-context tasks? Preprint, arXiv:2405.15318

    2024 arXiv

  17. [25]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. https://huggingface.co/datasets/cerebras/SlimPajama-627B SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    2023

  18. [26]

    Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung-Woo Ha, and Jinwoo Shin. 2023. Hierarchical context merging: Better long context understanding for pre-trained llms. In The Twelfth International Conference on Learning Representations

    2023

  19. [27]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  20. [28]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539--554

    2022

  21. [29]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. https://arxiv.org/abs/1706.03762 Attention is all you need . Preprint, arXiv:1706.03762

    2023 arXiv

  22. [30]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. https://arxiv.org/abs/2309.17453 Efficient streaming language models with attention sinks . Preprint, arXiv:2309.17453

    2024 arXiv

  23. [31]

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, S...

    2024 doi

  24. [32]

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. https://doi.org/10.48550/arxiv.2310.03025 Retrieval meets Long Context Large Language Models . arXiv. Experimental

    2023 doi arXiv

  25. [33]

    Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. 2024. Think: Thinner key cache by query-driven pruning. arXiv preprint arXiv:2407.21018

    2024 arXiv

  26. [34]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  27. [35]

    Xinhao Yao, Hongjin Qian, Xiaolin Hu, Gengze Xu, Yong Liu, Wei Liu, Jian Luan, and Bin Wang. 2024. Theoretical insights into fine-tuning attention mechanism: Generalization and optimization. arXiv preprint arXiv:2410.02247

    2024 arXiv

  28. [36]

    Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. 2024. https://arxiv.org/abs/2407.09014 Compact: Compressing retrieved documents actively for question answering . Preprint, arXiv:2407.09014

    2024 arXiv

  29. [37]

    Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. 2024 a . Soaring from 4k to 400k: Extending llm's context with activation beacon. arXiv preprint arXiv:2401.03462

    2024 arXiv

  30. [38]

    Peitian Zhang, Ninglu Shao, Zheng Liu, Shitao Xiao, Hongjin Qian, Qiwei Ye, and Zhicheng Dou. 2024 b . https://arxiv.org/abs/2404.19553 Extending llama-3's context ten-fold overnight . Preprint, arXiv:2404.19553

    2024 arXiv

  31. [39]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024 c . https://doi.org/10.18653/V1/2024.ACL-LONG.814 nftybench: Extending long context evaluation beyond 100k tokens . In Proceedi...

    2024 doi

  32. [40]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. https://arxiv.org/abs/2306.14048 H _2 o: Heavy-hitter oracle for efficient generative inference of lar...

    2023 arXiv

  33. [41]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2024 arXiv

  34. [42]

    Yujia Zhou, Yan Liu, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Zheng Liu, Chaozhuo Li, Zhicheng Dou, Tsung-Yi Ho, and Philip S Yu. 2024. Trustworthiness in retrieval-augmented generation systems: A survey. arXiv preprint arXiv:2409.10102

    2024 arXiv

  35. [43]

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zhicheng Dou, and Ji-Rong Wen. 2024. https://arxiv.org/abs/2308.07107 Large language models for information retrieval: A survey . Preprint, arXiv:2308.07107

    2024

  36. [44]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  37. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.