Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By encoding each context separately and caching its key-value states, APE aligns the parallel attention distribution with sequential encoding using a shared prefix, a lower attention temperature, and a scaling factor, preserving 98% (RAG)…

desk verdict Useful training-free recipe for parallel encoding, but the central equation is wrong as printed; needs a corrected derivation before the claims are trustworthy. read the letter →

arxiv 2502.05431 v2 pith:WFXRGIQ7 submitted 2025-02-08 cs.LG cs.AI

Xinyu Yang , Tianqi Chen , Beidi Chen This is my paper · ORCID
classification cs.LGcs.AI
keywords parallelencodingcontext-augmentedgenerationretrieval-augmentedin-contextlearningKVcacheattentionsinktemperaturelong-contextprefill
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that context-augmented generation — retrieval-augmented generation and in-context learning — can encode every retrieved text in parallel, cache its key-value states, and stitch the cached states together at query time, without the accuracy collapse that previously forced either full sequential re-encoding or fine-tuning. The paper first shows why naive parallel encoding fails: each independently encoded context duplicates the anomalous first-token states, and duplicated neighbouring tokens flatten the softmax toward a uniform distribution that dilutes the relevant information. It then proposes APE, Adaptive Parallel Encoding, three training-free adjustments — a shared prefix, a lowered attention temperature, and a scaling factor on the context attention scores — that realign the parallel attention distribution with the sequential one. On RAG and ICL benchmarks this preserves 98% and 93% of sequential encoding performance respectively, beats naive parallel encoding by 3.6% and 7.9%, and cuts prefilling time by 28x for a 128K-token context, an end-to-end 4.5x speedup. If correct, cached context databases become practical: contexts can be added, removed, or swapped without re-encoding, and far more context fits in a fixed window, which is precisely the regime where long-context prefill, not generation, dominates latency.

What carries the argument

The central mechanism is a hierarchical, two-level form of softmax attention in which each context $C_i$ is first reduced to an attention head $V^h_{C_i}$ together with a LogSumExp scalar $A^h_{C_i}$, and the query then merges contexts at the second level, $O = \mathrm{Softmax}([A^h_{C_1},\dots,A^h_{C_N}, A^h])\,[V^h_{C_1},\dots,V^h_{C_N},V^h]$. APE's three corrections act on this hierarchy: a shared prefix (the system prompt, or newlines when none exists) removes the duplicated abnormal initial states; a temperature $T<1$ sharpens the intra-context softmax so the query attends to the relevant tokens; and a scale $S<1$ rescales each context's LogSumExp to cancel the magnitude inflation that temperature introduces. This hierarchical form is what makes merged parallel caches computable with flash attention at negligible extra cost, and the paper notes it extends naturally to tree-structured caches where shared sub-contexts are merged recursively.

What would settle it

Run APE with a fixed temperature $T$ and scaling factor $S$ across LLaMA-3.1-8B-Instruct and Gemma-2-9b-it on the same eight-shot GSM8K prompts and measure the accuracy margin over naive parallel encoding; the premise predicts a consistent positive margin, while the paper's own appendix shows Gemma's first-token key states align more weakly across inputs, so a vanishing or negative margin on the weakly aligned model would show that the corrections depend on the attention sink's directional anchoring rather than on the temperature and scaling operations themselves.

Watch

Extended reading notes

Core claim

The central discovery is that the key and value states of independently encoded contexts are already geometrically comparable: because of the attention sink, the initial-token key states point in nearly the same direction for different inputs, and key and value states at later positions keep similar directions and magnitudes across contexts. This means naive parallel encoding is not fundamentally broken — the KV states can be merged into one sequence — and its remaining errors are a localized distribution mismatch rather than a representational failure. The mismatch shows up as anomalous attention mass at the first few tokens of each context and over-uniform attention among neighbouring tokens, and the paper shows these are corrected by three inference-only operations: prepending a shared prefix so the abnormal initial states appear only once, lowering the attention temperature $T<1$ to sharpen the softmax, and applying a scaling factor $S<1$ to offset the inflated LogSumExp of the context scores. With these operations the paper reports 98% and 93% preservation of sequential accuracy on RAG and ICL, 3.6% and 7.9% gains over parallel encoding, and up to a 28x prefill reduction at 128K tokens.

Load-bearing premise

The method only works if separately encoded contexts produce key and value states with nearly identical directions and magnitudes, anchored by the fixed initial token, so that merging them is legitimate — and the fixed shared prefix, temperature, and scaling factor must also keep working for new queries without retuning.

Editorial extensions

If this is right

  • Cached contexts become freely recombinable: inserting, deleting, replacing, or swapping one retrieved passage costs nothing at inference, because only that passage needs re-encoding.
  • Position reuse packs far more context into the same window, and in many-shot RAG and ICL the paper finds this matches or exceeds sequential encoding despite comparing equal inputs.
  • Prefill stops being the bottleneck: at 128K tokens the paper reports prefill time falling by 28x, so a workload that spent more than half its time encoding context spends under 10% of it there.
  • The corrections compose with fast-attention serving engines, so the accuracy recovery does not add a latency tax that cancels the speedup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The position-reuse property gives a controlled way to test how much of long-context degradation is positional: if APE cures 'lost in the middle' simply by making every chunk equally close to the query, then part of that failure is an artifact of sequential position assignment rather than a hard capacity limit.
  • The temperature and scaling factor are hand-tuned per task and model, and the paper reports they are its main sensitivity; an automatic calibration from the cached contexts' own LogSumExp statistics would remove the operational barrier the limitations section flags.
  • The hierarchical merge formula suggests treating overlapping retrieved passages as a tree with shared sub-contexts encoded once, which would cut both storage and prefill for collections with reused passages — an extension the paper sketches but does not evaluate.
  • If APE holds on more models, the long-context fine-tuning race loses some of its point for retrieval workloads: database-style pre-caching plus parallel merge could extend effective context beyond the trained window without any training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Adaptive Parallel Encoding (APE), a training-free method for context-augmented generation that precomputes and caches KV states of contexts independently and then, at inference time, combines them with the query. To compensate for the distributional mismatch between parallel and sequential encoding, APE prepends a shared prefix, lowers the attention temperature, and applies a scaling factor to the LogSumExp of per-context attention scores. The authors report evaluations on RAG and ICL benchmarks, including ChatRAG-Bench, LongBench, LOFT, and CRAG, and latency measurements showing large prefilling speedups, with claimed accuracy preservation of 98% and 93% relative to sequential encoding on same-input RAG and ICL settings.

Significance. If the method is sound, APE would be a practically important training-free technique: it would turn independently precomputed per-context KV caches into a usable long-context CAG mechanism, avoiding re-encoding and giving large prefilling speedups. The paper's strengths are its systematic analysis of KV-state geometry across layers and models, a component-wise ablation, multiple benchmark evaluations, and a code snippet that clarifies the intended implementation. The claims that same-input APE preserves most sequential encoding accuracy and outperforms naive parallel encoding are meaningful and, if confirmed, would justify the contribution. However, the central attention formula is specified inconsistently across Section 4.4 and Appendix C, and the main LongBench comparison conflates the encoding method with a much larger context budget; these issues currently prevent full confidence in the stated results.

major comments (3)
  1. [Section 4.4, Eq. (5); Appendix C.3 (Eqs. 13–15); Appendix C.4] The paper does not specify APE unambiguously because Eq. (5), the derivation in Appendix C.3, and the code in Appendix C.4 implement different normalization rules. In Eq. (5), each context token's unnormalized score is multiplied by the global sum over all contexts raised to the power S−1, so the total context weight is (Σ_{i=1}^N Σ_{j=1}^{l_{C_i}} a'_{C_i,j})^S and the denominator contains exactly that global term. In contrast, the hierarchical formula in Eq. (12) and the code snippet in Section C.4 compute a per-context LogSumExp, scale it by S, and then softmax over contexts; this yields per-token weights proportional to exp(a'_{C_i,j}) · (Σ_{t∈C_i} exp(a'_{C_i,t}))^{S−1} with denominator Σ_i (Σ_{t∈C_i} exp(a'))^S + Σ_t exp(a_t). Equation (15) uses the per-context sum in the numerator but a global sum in the denominator, and is algebraically inconsistent with Eq. (14) and with Eq. (5) unless N=1 or S=1. Since the experimental numbers are produced by whichever rule is actually implemented, the manuscript's central method is not reproducibly defined. Please unify the definition, state which rule was used in all experiments, and correct the appendix derivation.
  2. [Section 5.1.2, Table 2] The LongBench comparison does not hold the input budget fixed. The sequential RAG baseline uses C200×20, i.e., 20 chunks of 200 words each (roughly 4,000 words), while APE uses C4000×20, i.e., 20 chunks of 4,000 words each (roughly 80,000 words), a 20× difference in context length. The claim that APE 'outperforms sequential RAG baselines by 3.3%' therefore conflates the encoding method with a much larger context budget. This is not a same-input comparison and cannot be attributed solely to the alignment corrections. Please separate the two effects by reporting APE with the same retrieved chunks as the sequential baseline (as is done in Table 1) and sequential encoding with the extended chunks, or else explicitly label the comparison as 'with more context' and avoid claiming an encoding-method victory from this table.
  3. [Section 3.2 and Appendix B] The method's core premise is that key and value states from independently encoded contexts are sufficiently aligned in direction and magnitude to be merged directly, because of the attention sink. The appendix itself shows that this alignment is much weaker for Gemma-2-9b-it than for the LLaMA models: Figures 11(d) and 12(d) report substantially lower inter-sample cosine similarities for initial key and value states. Table 2 nevertheless shows APE improving Gemma-2-9b-it results, which is encouraging, but the paper does not explain how the three corrections compensate for weaker initial-state alignment. Please either analyze this case or state explicitly that the method is validated only for models with strong attention-sink alignment; without such a statement, the claimed generality of the alignment premise is not established.
minor comments (5)
  1. [Appendix C.3, Eq. (13) and Eq. (15)] Equation (13) contains an index error: the exponent inside the sum over n and t should use Q k_{C_n,t}^T, not Q k_{C_i,t}^T, and the same issue appears in the denominator of Eq. (15). Please correct the indices to make the global versus per-context sums unambiguous.
  2. [Section 4.4] The sentence 'We use S·T instead of S as the scaling factor to simplify our search' is confusing, because Equations (5) and (11) use S and T as separate hyperparameters. Please state clearly whether the searched quantity is the product S·T applied to the LSE, or both S and T are tuned independently.
  3. [Tables 1 and Figure 9] The accuracy results are reported without variance or number of seeds. In particular, the Table 1 average gaps of 0.51, 0.92, and 1.14 points are close to typical run-to-run noise for language-model evaluation; please report standard deviations or multiple seeds, or justify determinism of the evaluation protocol.
  4. [Abstract and Section 5.4] The abstract says APE reduces prefilling time by 28× without specifying the baseline; Section 5.4 states this 28× reduction is compared to MInference, while the 4.5× inference speedup is compared to sequential encoding. Please make the baseline explicit in the abstract or in the efficiency section to avoid misleading the reader.
  5. [Table 4] The 'Score' column in Table 4 is not defined. Please give the formula used to combine Accuracy, Hallucination, and Missing, or remove the score if it is a simple aggregation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: APE's claimed gains are empirical, benchmarked against external baselines, with hyperparameters tuned on a validation set and the attention-sink premise independently supported.

full rationale

APE's central claims are not defined into existence. The three alignment modifications (shared prefix, temperature T, scaling factor S) are introduced heuristically (Sections 4.1-4.3), and the final formula in Section 4.4 is a re-arrangement of softmax attention, not an identity that guarantees the reported accuracy. Hyperparameters are selected by greedy search on a small validation set ('For the choice of hyperparameters, we conduct a greedy search over a small validation set'), then evaluated on held-out benchmark tasks; this is standard tuning rather than fitting the target result. The attention-sink justification cites Xiao et al. 2023, which shares author Beidi Chen, but that prior result is externally established and, more importantly, the paper independently measures KV-state cosine similarity and magnitude in Figures 3-5 and Appendix B, so the self-citation is not the load-bearing evidence. The efficiency numbers (4.5x end-to-end, 28x prefill reduction) are measured latency comparisons against sequential encoding and MInference on an H100, following directly from pre-caching but not circular because they are empirical. Two caveats belong to correctness, not circularity: Section 4.4 Eq. 5 as printed does not normalize to a probability-weighted average when S is not 1 (context tokens contribute A^S in the numerator while the denominator uses A*S), and Appendix C.3 Eq. 15 repeats a per-context versus global sum mismatch. Section 8 also concedes sensitivity to T and S. These affect reproducibility and robustness, not whether the method reduces to its own assumptions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

APE's contribution is entirely input-side: it introduces three per-task hyperparameters (T, S, shared-prefix length) and depends on the empirical attention-sink and KV-comparability premise established in Section 3.2. It postulates no new physical or model entities, so the invented-entity ledger is empty.

free parameters (3)
  • Attention temperature T = Per-task/model, searched in [0.1, 1.0] step 0.1; e.g., T=0.2 shown in Figure 7
    Adjusted per task via greedy validation search to sharpen attention (Section 4.2, 4.4).
  • Scaling factor S = Searched jointly via S*T in [0.1, 1.0] step 0.1; no final per-task values reported
    Applied to context LogSumExp to compensate for temperature-induced magnitude shift (Section 4.3, 4.4).
  • Shared prefix length = Starts with two newline characters; extended by 10, 20, or 40 tokens via validation search
    Chosen per task/model so the first-token anomaly appears only once (Section 4.1).
assumptions (4)
  • standard math Standard softmax attention: O = Softmax(QK^T/sqrt(d))V
    Used throughout; APE modifies the temperature and normalization of this operation.
  • domain assumption Attention sink: the first token receives disproportionately high attention in all evaluated LLMs
    Invoked in Section 3.2 and 4.1 to justify the shared prefix and KV comparability.
  • domain assumption KV states from independent contexts are comparable and mergeable because key states share directions and magnitudes across inputs
    Empirical claim from Section 3.2 (Figures 3-5) underpins the whole parallel-encoding design; not derived.
  • ad hoc to paper The three residual misalignments (first-token anomaly, local-position bias, LSE magnitude shift) are the only significant differences between parallel and sequential attention
    Section 4 assumes these three fixes suffice; no analysis of higher-order interactions or task-dependent failure modes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding." pith.science (2026). https://pith.science/paper/WFXRGIQ7

@misc{pith2026250205431,
  author       = {Pith},
  title        = {Pith review of: APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WFXRGIQ7}},
  note         = {Machine review of arXiv:2502.05431}
}
abstract

Context-augmented generation (CAG) techniques, including RAG and ICL, require the efficient combination of multiple contexts to generate responses to user queries. Directly inputting these contexts as a sequence introduces a considerable computational burden by re-encoding the combined selection of contexts for every request. To address this, we explore the promising potential of parallel encoding to independently pre-compute and cache each context's KV states. This approach enables the direct loading of cached states during inference while accommodating more contexts through position reuse across contexts. However, due to misalignments in attention distribution, directly applying parallel encoding results in a significant performance drop. To enable effective and efficient CAG, we propose Adaptive Parallel Encoding ($\textbf{APE}$), which brings shared prefix, attention temperature, and scaling factor to align the distribution of parallel encoding with sequential encoding. Results on RAG and ICL tasks demonstrate that APE can preserve 98% and 93% sequential encoding performance using the same inputs while outperforming parallel encoding by 3.6% and 7.9%, respectively. It also scales to many-shot CAG, effectively encoding hundreds of contexts in parallel. Efficiency evaluation shows that APE can achieve an end-to-end 4.5$\times$ speedup by reducing 28$\times$ prefilling time for a 128K-length context.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Weak-to-Strong GraphRAG: Aligning Weak Retrievers with Large Language Models for Graph-based Retrieval Augmented Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    ReG refines weak graph-retriever supervision with LLM-selected reasoning chains and reorganizes retrieved triples into coherent evidence chains, improving KGQA accuracy, data efficiency, and reasoning token efficiency.

Reference graph

Works this paper leans on

37 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [4]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508 ,

    arXiv

  3. [5]

    Memory transformer

    Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprint arXiv:2006.11527,

    arXiv 2006

  4. [7]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021a. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry ...

    arXiv

  5. [8]

    Mention Memory: incorporating textual knowledge into Transformers through entity mention attention

    Michiel De Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Fei Sha, and William Cohen. Mention memory: incorporating textual knowledge into transformers through entity mention attention. arXiv preprint arXiv:2110.06176 ,

    work page Pith review arXiv

  6. [11]

    The faiss library.arXiv preprint arXiv:2401.08281 ,

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazar´ e, Maria Lomeli, Lucas Hosseini, and Herv´ e J´ egou. The faiss library.arXiv preprint arXiv:2401.08281 ,

    arXiv

  7. [12]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  8. [13]

    Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model

    Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. arXiv preprint arXiv:1906.01749 ,

    arXiv 1906

Show all 37 references
  1. [15]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang

    https://zenodo.org/records/12608602. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 ,

  2. [16]

    Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture

    14 Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al. Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture. arXiv preprint arXiv:2401.08406 ,

    arXiv

  3. [17]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020a. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, ...

    2009 arXiv

  4. [19]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  5. [20]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490 , 2024a....

    arXiv

  6. [21]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551 ,

    arXiv

  7. [22]

    Can long-context language models subsume retrieval, rag, sql, and more? arXiv preprint arXiv:2406.13121 ,

    Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, S´ ebastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al. Can long-context language models subsume retrieval, rag, sql, and more? arXiv preprint arXiv:2406.13121 ,

    arXiv

  8. [23]

    Towards general text embeddings with multi-stage contrastive learning

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281 ,

    arXiv

  9. [24]

    Focusllm: Scaling llm’s context by parallel decoding

    Zhenyu Li, Yike Zhang, Tengyu Pan, Yutao Sun, Zhichao Duan, Junjie Fang, Rong Han, Zixuan Wang, and Jianyong Wang. Focusllm: Scaling llm’s context by parallel decoding. arXiv preprint arXiv:2408.11745 ,

    arXiv

  10. [25]

    Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor R¨ uhle, Yuqing Yang, Chin-Yew Lin, et al. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. arXiv preprint arXiv:2403.12968 ,

    arXiv

  11. [26]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409 ,

    arXiv

  12. [27]

    Parallel context windows for large language models

    15 Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. Parallel context windows for large language models. arXiv preprint arXiv:2212.10947,

    arXiv

  13. [28]

    Block-attention for efficient rag

    East Sun, Yan Wang, and Lan Tian. Block-attention for efficient rag. arXiv preprint arXiv:2409.15355 ,

    arXiv

  14. [29]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L´ eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´ e, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 ,

    arXiv

  15. [31]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 ,

    arXiv

  16. [32]

    Crag–comprehensive rag benchmark

    Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, et al. Crag–comprehensive rag benchmark. arXiv preprint arXiv:2406.04744 ,

    arXiv

  17. [33]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 ,

    arXiv

  18. [34]

    Long-context language modeling with parallel context encoding

    Howard Yen, Tianyu Gao, and Danqi Chen. Long-context language modeling with parallel context encoding. arXiv preprint arXiv:2402.16617,

    arXiv

  19. [35]

    Qdrant, 2024.https://qdrant.tech/

    Andr´ e Zayarni, Andrey Vasnetsov, et al. Qdrant, 2024.https://qdrant.tech/. 16 Appendix A Detailed Experimental Setups for Section 3.1 RAG. We select four tasks that require processing multiple input documents from the LongBench dataset (Bai et al., 2023), including HotpotQA ...

    2024

  20. [36]

    (2017), and MMLU (Hendrycks et al., 2020b)

    to evaluate the ICL ability of different encoding methods, involving GSM8K (Cobbe et al., 2021b), TriviaQA Joshi et al. (2017), and MMLU (Hendrycks et al., 2020b). In parallel encoding and CEPED, we will encode each example separately and input all the resulting KV states to Θ...

    2017

  21. [37]

    This flexibility allows each user query to be enriched with external knowledge organized in such structures, demonstrating APE’s capability to handle structured external data effectively. 21 (a) Parallel Cache Structure (b) Tree Cache Structure SharedPrefix Context1 Context2 C...

    2024

  22. [2017]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 ,

    arXiv

  23. [2018]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234 ,

    arXiv

  24. [2019]

    Entities as experts: Sparse memory access with entity supervision

    Thibault F´ evry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. Entities as experts: Sparse memory access with entity supervision. arXiv preprint arXiv:2004.07202 ,

    2004 arXiv

  25. [2020]

    Unsupervised dense information retrieval with contrastive learning

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118 ,

    arXiv

  26. [2021]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

    arXiv

  27. [2022]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian

    https://github.com/langchain-ai/langchain. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595 ,

    arXiv

  28. [2023]

    Many-shot in-context learning

    Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018 ,

    arXiv

  29. [2024]

    Reliable, adaptable, and attributable language models with retrieval

    Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen-tau Yih. Reliable, adaptable, and attributable language models with retrieval. arXiv preprint arXiv:2403.03187 ,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.