Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-grained gist-token context compression is nearly lossless for fuzzy long-context tasks but collapses on exact recall, with three named failure patterns and two mitigations.

desk verdict A genuinely useful empirical map of when gist-token compression works and fails, but the TIE weighting formula appears inverted relative to its stated goal, which needs a fix or a clear explanation before the paper is publishable. read the letter →

arxiv 2412.17483 v1 pith:VQ3UW5KG submitted 2024-12-23 cs.CL

Chenlong Deng , Zhisong Zhang , Kelong Mao , Shuaiyi Li , Xinting Huang , Dong Yu , Zhicheng Dou This is my paper · ORCID
classification cs.CL
keywords gisttokenscontextcompressionKVcachelong-contextlanguagemodelsfailurepatternsautoencodingtokenimportanceestimationsyntheticrecall
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether replacing full attention with a small set of learned gist tokens can be a true substitute for long-context processing, and answers with a qualified yes: for tasks that need fuzzy topical information—retrieval-augmented generation, long-document question answering, many-shot in-context learning—compression at low ratios is nearly lossless. The qualification is sharp: tasks that demand exact recall degrade dramatically, with synthetic recall falling to 40.6 versus 93.9 for full attention at compression ratio 4. The paper's contribution is a diagnosis as well as a measurement: probing shows gist representations lose information, and three named failure patterns ('lost by the boundary', 'lost if surprise', 'lost along the way') account for the degradation. It then shows that two training strategies, fine-grained autoencoding and segment-wise token importance estimation, recover a meaningful share of the lost performance. The practical conclusion is a map: gist compression is a viable trade for memory-heavy but topic-tolerant applications, and a known failure mode for exactness-critical ones.

What carries the argument

The central object is the fine-grained KV-cache gist architecture (Fine-KV), in which the input is split into fixed-length segments and one gist token is inserted evenly within each snippet, so every gist token is responsible for encoding a local span while attending to all previous compressed memory; gist tokens are learned special tokens that stand in for a block of natural tokens, and at compression ratio 4 one gist token replaces four raw tokens. The explanatory machinery is a probing autoencoder that trains a decoder to reconstruct the original token sequence from each gist token, converting compression loss into a measurable readout accuracy; the three failure patterns and both mitigation losses are anchored to this readout. The mitigations operate at the same level: fine-grained autoencoding adds a reconstruction loss per gist token, and segment-wise token importance estimation reweights each token's language-modeling loss by how much its prediction depends on compressed context.

What would settle it

Train the probing decoder for substantially more than 2K steps, or vary its capacity and input format, and check whether reconstruction accuracy at ratio 4 rises far above the reported 53.9% for the weak decoder and 77.3% for the strong decoder; if it approaches full-attention copy levels, the claim that gist representations are the bottleneck would need revision.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that fine-grained KV-cache gist compression—replacing each snippet of raw tokens with one evenly inserted learned gist token—can approach full-attention quality on tasks where the answer only needs rough topical overlap, including retrieval-augmented generation, long-document QA, and many-shot in-context learning, but collapses on tasks demanding exact reproduction: at compression ratio 4, synthetic recall drops from 93.9 to 40.6. Probing with an autoencoding readout shows the gist representations themselves are the bottleneck; at ratio 4 a strong decoder reconstructs only 77.3% of tokens and a weak decoder 53.9%. Three failure patterns explain the drop: information near segment boundaries is degraded, contextually surprising information is preferentially discarded when budgets are tight, and long exact strings that must be copied across multiple gist tokens are lost midway through generation. Two training strategies—fine-grained autoencoding with a weak decoder and segment-wise token importance estimation—reduce these failures, lifting the average score from 46.1 to 50.1 at compression ratio 4.

Load-bearing premise

The causal story that gist representations are the compression bottleneck rests on the assumption that a decoder trained for only 2K steps can faithfully read out whatever the gist tokens store; if that probe is under-trained or mismatched, the reported reconstruction accuracies would understate how much information compression actually preserves.

Editorial extensions

If this is right

  • At compression ratio 4, fine-grained KV-cache gist models stay within about one point of full attention on RAG, long-document QA, and many-shot ICL, so roughly 75% memory savings are available without sacrificing these workloads.
  • Exact-recall tasks such as synthetic recall and reranking remain the weak points, so production deployments that need verbatim retrieval should keep a full-attention fallback.
  • The three failure patterns yield concrete diagnostics: monitoring perplexity near segment starts, testing off-theme needles, and checking long exact-string copy accuracy will reveal whether a compressed model is likely to fail.
  • Fine-grained autoencoding and segment-wise token importance estimation are complementary; together they raise the ratio-4 average from 46.1 to 50.1, and the paper's supervised fine-tuning experiment shows long-distance supervision can further help the weakest task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the boundary effect appears at every segment start, the same position-wise perplexity spike is likely to show up in any block-wise memory scheme, not only gist tokens; re-running the paper's per-position perplexity plot on token-eviction methods would test whether this is gist-specific or segment-wise.
  • Editorial extension: the 'lost if surprise' pattern suggests a training intervention the paper does not try—deliberately scattering surprising needles through training segments and upweighting their reconstruction loss—which would separate a corpus-statistics explanation from an architectural ceiling.
  • Editorial extension: the 2K-step probe leaves open whether gist representations are failing to store information or merely failing to expose it to a shallow decoder; extending the probe to convergence would move the bottleneck attribution from the representation to the readout, or back.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents a systematic empirical study of gist-token-based context compression for LLMs. It proposes a two-dimensional taxonomy (memory location and gist granularity), evaluates three resulting architectures on Llama3.1-8B and Qwen2-7B across language modeling, weak-context, and long-context tasks, and reports that fine-grained KV-cache compression (Fine-KV) is nearly lossless on fuzzy tasks such as RAG and long-document QA but degrades sharply on exact-recall tasks such as synthetic recall. The paper further identifies three failure patterns: lost by the boundary, lost if surprise, and lost along the way. It then proposes two mitigation strategies, fine-grained autoencoding and segment-wise token importance estimation (TIE), and reports consistent average gains, especially at low compression ratios.

Significance. If validated, the paper would make a useful empirical contribution: the task battery is broad, the architecture comparison is carried out under a unified training recipe on two base models, and the failure-pattern taxonomy is a valuable qualitative framework for future work on gist-based compression. The paper is also careful to qualify 'near-lossless' by task and includes explicit limitations. However, the central explanation of the TIE method is mathematically inverted as written, the probing experiment that grounds the failure taxonomy is not shown to be converged, and no variance or significance information is reported anywhere. These issues are load-bearing for the paper's claims, so the manuscript needs major revision before the results can be accepted.

major comments (4)
  1. [§5.1, TIE formula] The displayed TIE formula defines Diff(xi) = min(log Pθ(xi|xseg_<i) / Pθ(xi|xfull_<i), γ) and sets wi ∝ e^{Diff(xi)}. Since xseg_<i excludes the previous compressed segments, for a model trained on full context one expects Pθ(xi|xfull_<i) ≥ Pθ(xi|xseg_<i) for most tokens, making the log-ratio ≤ 0 and more negative when the token depends more on the compressed context. Consequently e^{Diff(xi)} is smallest precisely for the tokens that §5.1 says should receive the most learning effort. As written, the formula implements the opposite of the stated objective, and the reported gains (e.g., synthetic recall +13.7 at ratio 4 in Table 5) cannot be attributed to the described weighting. The authors must correct the formula (e.g., using log(Pfull/Pseg) or an equivalent monotone transformation), clarify the role and value of γ, and re-validate the empirical claims with the corrected definition.
  2. [§4.1, probing experiment] The conclusion that gist representations are the compression bottleneck rests on reconstruction accuracies from probing decoders trained for only 2K steps (Table 3). The paper does not report learning curves, convergence checks, or a control showing that the same decoder can reconstruct uncompressed token sequences. Without such evidence, the low reconstruction accuracies (53.9% at ratio 4, 5.1% at ratio 32) may understate what the gist tokens actually store, weakening the causal link between gist representations and the three failure patterns. Please report convergence behavior, train for substantially more steps or with early stopping, and include a decoder-capacity control.
  3. [All experimental tables] Every performance table reports a single point estimate with no error bars, repeated seeds, or significance tests. Many claimed gains are small (e.g., Table 5: +0.3 on RAG and +0.5 on LongQA at ratio 4), and some are negative at higher ratios (e.g., TIE on Code at ratio 16 is -1.1). Without variance estimates, the reader cannot determine whether the headline comparisons among architectures or the improvements from AE/TIE are robust. Please add multiple seeds or, at minimum, a significance analysis for the central comparisons in Tables 2 and 5.
  4. [§3.2 and Table 2] The compression models are continued-trained on 3B tokens, and Table 2 includes a 'Full Attention, Finetune' row as a matched control. However, the text consistently compares compression results against the base 'Full Attention' row when making claims such as 'near-lossless performance compared to the full attention model.' Since continued training itself changes performance (Full Attention Finetune averages 54.7 vs 55.6 for the base model), the matched finetuned full-attention baseline should be used when isolating the effect of compression. Please make the comparison against this control explicit and adjust any claims that rely on the base model as the reference.
minor comments (5)
  1. [§1, contributions list] The second bullet contains a duplicated 'that' ('We show that that gist-based models achieve...'); please fix the typo.
  2. [§2] The text contains the typo 'pervious work' in the first paragraph; it should read 'previous work.'
  3. [Table 5 vs Table 2] Table 5 lists the Fine-KV baseline at ratio 4 as 62.0 for Code, while Table 2 reports 63.0 for the same configuration; one of these values is inconsistent and should be corrected.
  4. [§5.1] The clipping threshold γ in the TIE formula is never defined; please state its range, default value, and whether it is tuned per task or per compression ratio.
  5. [Figure 4] The caption says 'When k is a multiple of 2048, the model will generate near the boundary,' but the figure's x-axis labels are in units of k; please clarify how the segment boundaries (at multiples of 2048) relate to the plotted k values and to the generation start positions.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: empirical results are measured against external benchmarks, and the proposed training objectives are not fitted to test outputs; the self-cited Activation Beacon architecture is evaluated rather than assumed, and the TIE sign issue is a correctness concern, not a circular step.

full rationale

The paper's load-bearing claims are empirical comparisons against external benchmarks (PG19, ProofPile, CodeParrot, RULER, infinityBench, MMLU-Pro, and others), not consequences of its own definitions or fitted parameters. The Fine-KV architecture is taken from Activation Beacon (Zhang et al., 2024a), a paper sharing corresponding author Zhicheng Dou, so a self-citation exists. However, the present paper re-trains and evaluates that architecture rather than invoking the citation as proof, and the headline findings (near-lossless performance on fuzzy tasks, sharp degradation on synthetic recall) are measured against independent full-attention baselines. The probing study in Section 4.1 is a diagnostic measurement: training a decoder for 2K steps and reading out reconstruction accuracy is not circular, though it may understate what gist tokens actually store if the probe is undertrained. The three failure patterns are induced from controlled experiments (perplexity-by-position, needle insertion, UUID copying), not derived from the paper's own conclusions. The proposed mitigations are training objectives, not fitted test predictors: fine-grained autoencoding adds an auxiliary reconstruction loss, and TIE reweights the next-token loss using the model's own probabilities. Neither uses test-set performance to define its objective. The TIE formula in Section 5.1 does contain a sign inconsistency with its stated rationale: tokens most dependent on compressed context receive the smallest e^Diff weight, so the reported gains are hard to attribute to the described formula. This is a correctness or implementation concern, not a circular reduction; the claim is not made true by construction, and the formula does not reduce to the paper's input assumptions. No uniqueness theorem from the authors' prior work is invoked, no ansatz is hidden behind a citation, and no known result is renamed as a derivation. The only circularity-adjacent issue is the self-citation of Activation Beacon as the central architecture under study, which is a minor conflict of interest but not load-bearing circularity. Accordingly, the paper is self-contained against external benchmarks and receives a low circularity score of 2.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on the chosen base models, training recipe, and benchmark tasks. The most fragile element is the probing assumption, which underpins the bottleneck explanation. The TIE method introduces a hyperparameter gamma that is not reported. No new entities are invented.

free parameters (6)
  • segment length L = 2048
    Sets the granularity of compression; chosen by hand, not fitted. Affects boundary effects.
  • training context length = 16384
    Maximum training sequence length; chosen by hand.
  • number of sink tokens = 4
    Added for training stability, chosen by hand.
  • TIE clipping threshold gamma
    The Diff(x_i) is clipped to gamma, but the value is not reported in the main text; likely set in appendix, representing a free hyperparameter.
  • autoencoding decoder layers = 1
    Single-layer transformer decoder chosen based on prior work on weak decoders.
  • probe training steps = 2000
    The probing decoder is trained for 2K steps; a fixed budget that affects measured reconstruction accuracy.
assumptions (5)
  • domain assumption The base models Llama3.1-8B and Qwen2-7B are representative of large language models for studying context compression.
    Section 3.1 selects these two models and generalizes conclusions from them.
  • domain assumption Continue-training on 2B tokens of upsampled SlimPajama with random compression ratios teaches gist compression without confounding SFT.
    Section 3.1 and Appendix A describe this recipe; the paper assumes this suffices to learn compression.
  • ad hoc to paper The probing decoder can faithfully decode the content of gist tokens after 2K training steps.
    Section 4.1 uses reconstruction accuracy from the probe as evidence of a compression bottleneck; if the probe is weak, the bottleneck may be overstated.
  • ad hoc to paper Token importance can be estimated by the log-probability difference between segment-only and full-context conditioning.
    Section 5.1 defines Diff(x_i) and uses it to set loss weights; this is a modeling choice specific to this paper.
  • domain assumption The 'lost if surprise' experiment's subject replacement ('Mr. Tree') isolates surprise without changing the difficulty of retrieving the needle.
    Section 4.2 and Table 13 replace the subject to create surprising needles; retrieval difficulty could differ independently of compression.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression." pith.science (2026). https://pith.science/paper/VQ3UW5KG

@misc{pith2026241217483,
  author       = {Pith},
  title        = {Pith review of: A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VQ3UW5KG}},
  note         = {Machine review of arXiv:2412.17483}
}
read the original abstract

In this work, we provide a thorough investigation of gist-based context compression methods to improve long-context processing in large language models. We focus on two key questions: (1) How well can these methods replace full attention models? and (2) What potential failure patterns arise due to compression? Through extensive experiments, we show that while gist-based compression can achieve near-lossless performance on tasks like retrieval-augmented generation and long-document QA, it faces challenges in tasks like synthetic recall. Furthermore, we identify three key failure patterns: lost by the boundary, lost if surprise, and lost along the way. To mitigate these issues, we propose two effective strategies: fine-grained autoencoding, which enhances the reconstruction of original token information, and segment-wise token importance estimation, which adjusts optimization based on token dependencies. Our work provides valuable insights into the understanding of gist token-based context compression and offers practical strategies for improving compression capabilities.

Figures

Figures reproduced from arXiv: 2412.17483 by the authors.

Figure 1
Figure 1. Overview of gist token-based context compression architectures. Long texts are segmented for compres [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparisons of different compression methods on perplexity evaluation for language modeling. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Average Perplexity of tokens in different positions among segments. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance on different tasks while truncat [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Performance on the 32-digit uuid recall task. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  2. GEM: Empowering LLM for both Embedding Generation and Language Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    GEM fine-tunes decoder-only LLMs with bottleneck special tokens and a mixed next-token prediction and contrastive objective, giving them text-embedding ability with only a modest MMLU drop.

  3. Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention

    cs.CL 2025-05 conditional novelty 6.0 of 10

    HyCo2 combines soft global compression with hard local token selection, reporting QA performance near uncompressed retrieval while cutting context tokens by about 88.8%.

Reference graph

Works this paper leans on

58 extracted references · 13 canonical work pages · cited by 3 Pith papers

  1. [1]

    Joshua Ainslie, James Lee - Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \' o n, and Sumit Sanghai. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.298 GQA: training generalized multi-query transformer models from multi-head checkpoints . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Si...

    doi:10.18653/v1/2023.emnlp-main.298 2023

  2. [2]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. https://arxiv.org/abs/2004.05150 Longformer: The long-document transformer . CoRR, abs/2004.05150

    arXiv 2020

  3. [3]

    William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan - Kelley. 2024. https://doi.org/10.48550/ARXIV.2405.12981 Reducing transformer key-value cache size with cross-layer attention . CoRR, abs/2405.12981

    arXiv doi:10.48550/arxiv.2405.12981 2024

  4. [4]

    Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. 2022. https://doi.org/10.48550/ARXIV.2207.06881 Recurrent memory transformer . CoRR, abs/2207.06881

    arXiv doi:10.48550/arxiv.2207.06881 2022

  5. [5]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. https://doi.org/10.48550/ARXIV.2306.15595 Extending context window of large language models via positional interpolation . CoRR, abs/2306.15595

    arXiv doi:10.48550/arxiv.2306.15595 2023

  6. [6]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. https://openreview.net/forum?id=6PmJoRfdaK Longlora: Efficient fine-tuning of long-context large language models . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    work page 2024

  7. [7]

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.232 Adapting language models to compress contexts . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pages 3829--3846. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.232 2023

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . CoRR, abs/2110.14168

    arXiv 2021

Show all 58 references
  1. [9]

    https://huggingface.co/codeparrot/codeparrot https://huggingface.co/codeparrot/codeparrot

    CodeParrot. https://huggingface.co/codeparrot/codeparrot https://huggingface.co/codeparrot/codeparrot

  2. [10]

    DeepSeek - AI. 2024. https://doi.org/10.48550/ARXIV.2405.04434 Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model . CoRR, abs/2405.04434

    2024 doi arXiv

  3. [11]

    Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. 2024. https://doi.org/10.48550/ARXIV.2410.23771 What is wrong with perplexity for long-context language modeling? CoRR, abs/2410.23771

    2024 doi arXiv

  4. [12]

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. https://openreview.net/forum?id=TaAqeo7lUh Data engineering for scaling language models to 128k context . In Forty-first International Conference on Machine Learning, ICML 2024, V...

    2024

  5. [13]

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2024. https://doi.org/10.48550/ARXIV.2410.02660 How to train long-context language models (effectively) . CoRR, abs/2410.02660

    2024 doi

  6. [14]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023. https://doi.org/10.48550/ARXIV.2312.10997 Retrieval-augmented generation for large language models: A survey . CoRR, abs/2312.10997

    2023 doi arXiv

  7. [15]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024 a . https://openreview.net/forum?id=uNrFpDPMyo Model tells you what to discard: Adaptive KV cache compression for llms . In The Twelfth International Conference on Learning Representations, ICLR...

    2024

  8. [16]

    Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si - Qing Chen, and Furu Wei. 2024 b . https://openreview.net/forum?id=uREj4ZuGJE In-context autoencoder for context compression in a large language model . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vie...

    2024

  9. [17]

    Cheng - Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. https://doi.org/10.48550/ARXIV.2404.06654 RULER: what's the real context size of your long-context language models? CoRR, abs/2404.06654

    2024 doi arXiv

  10. [18]

    Abdi, Dongsheng Li, Chin - Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin - Yew Lin, Yuqing Yang, and Lili Qiu. 2024 a . https://doi.org/10.48550/ARXIV.2407.02490 Minference 1.0: Accelerating pre-filling for long-context l...

    2024 doi arXiv

  11. [19]

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin - Yew Lin, Yuqing Yang, and Lili Qiu. 2024 b . https://doi.org/10.18653/V1/2024.ACL-LONG.91 Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression . In Proceedings of the 62nd A...

    2024 doi

  12. [20]

    Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. https://openreview.net/forum?id=rkgNKkHtvB Reformer: The efficient transformer . In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net

    2020

  13. [21]

    Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. https://doi.org/10.18653/V1/2022.FINDINGS-EMNLP.488 BOOKSUM: A collection of datasets for long-form narrative summarization . In Findings of the Association for Computational Lingui...

    2022 doi

  14. [22]

    Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.391 Compressing context to enhance inference efficiency of large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, ...

    2023 doi

  15. [23]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In The Twelfth International Conference on Learning Re...

    2024

  16. [24]

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. 2024 a . https://doi.org/10.48550/ARXIV.2405.14366 Minicache: KV cache compression in depth dimension for large language models . CoRR, abs/2405.14366

    2024 doi arXiv

  17. [25]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/a452a7c6c463e4ae8fbdc614c6e983e6-Abstract-Conference.html Scissorhands: Exploiting the per...

    2023

  18. [26]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024 b . https://openreview.net/forum?id=L057s2Rq8O KIVI: A tuning-free asymmetric 2bit quantization for KV cache . In Forty-first International Conference on Machine Le...

    2024

  19. [27]

    Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie - Yan Liu, and Arnold Overwijk. 2021. https://doi.org/10.18653/V1/2021.EMNLP-MAIN.220 Less is more: Pretrain a strong siamese encoder for dense text retrieval using a weak decoder . In Pro...

    2021 doi

  20. [28]

    Meta-Llama. 2024. https://doi.org/10.48550/ARXIV.2407.21783 The llama 3 herd of models . CoRR, abs/2407.21783

    2024 doi arXiv

  21. [29]

    Amirkeivan Mohtashami and Martin Jaggi. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/ab05dc8bf36a9f66edbff6992ec86f56-Abstract-Conference.html Random-access infinite context length for transformers . In Advances in Neural Information Processing Systems 36: Annual C...

    2023

  22. [30]

    Jesse Mu, Xiang Li, and Noah D. Goodman. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/3d77c6dcc7f143aa2154e7f4d5e22d68-Abstract-Conference.html Learning to compress prompts with gist tokens . In Advances in Neural Information Processing Systems 36: Annual Conferenc...

    2023

  23. [31]

    OpenAI. 2023. https://doi.org/10.48550/ARXIV.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    2023 doi arXiv

  24. [32]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. https://openreview.net/forum?id=wHBfxhZu1u Yarn: Efficient context window extension of large language models . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria,...

    2024

  25. [33]

    Hongjin Qian, Peitian Zhang, Zheng Liu, Kelong Mao, and Zhicheng Dou. 2024. https://doi.org/10.48550/ARXIV.2409.05591 Memorag: Moving towards next-gen RAG via memory-inspired knowledge discovery . CoRR, abs/2409.05591

    2024 doi arXiv

  26. [34]

    Guanghui Qin and Benjamin Van Durme. 2023. https://proceedings.mlr.press/v202/qin23a.html Nugget: Neural agglomerative embeddings of text . In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine...

    2023

  27. [35]

    Qwen-Team. 2024. https://doi.org/10.48550/ARXIV.2407.10671 Qwen2 technical report . CoRR, abs/2407.10671

    2024 doi arXiv

  28. [36]

    Rae, Anna Potapenko, Siddhant M

    Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. https://openreview.net/forum?id=SylKikSYDH Compressive transformers for long-range sequence modelling . In 8th International Conference on Learning Representations, ICLR 2020, Ad...

    2020

  29. [37]

    Noam Shazeer. 2019. https://arxiv.org/abs/1911.02150 Fast transformer decoding: One write-head is all you need . CoRR, abs/1911.02150

    2019 arXiv

  30. [38]

    Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. 2024. https://doi.org/10.48550/ARXIV.2405.05254 You only cache once: Decoder-decoder architectures for language models . CoRR, abs/2405.05254

    2024 doi arXiv

  31. [39]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Sch \" a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-ACL.824 Challenging big-bench tasks and whether chain-of-tho...

    2023 doi

  32. [40]

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020. https://arxiv.org/abs/2009.06732 Efficient transformers: A survey . CoRR, abs/2009.06732

    2020 arXiv

  33. [41]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. https://doi.org/10.48550/ARXIV.2406.01574 Mmlu-pro: A more r...

    2024 doi arXiv

  34. [42]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html Chain-of-thought prompting elicits reasoning...

    2022

  35. [43]

    Haoyi Wu and Kewei Tu. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.602 Layer-condensed KV cache for efficient inference of large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, ...

    2024 doi

  36. [44]

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. 2024 a . https://doi.org/10.48550/ARXIV.2402.04617 Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-...

    2024 doi arXiv

  37. [45]

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2024 b . https://doi.org/10.48550/ARXIV.2410.10819 Duoattention: Efficient long-context LLM inference with retrieval and streaming heads . CoRR, abs/2410.10819

    2024 doi arXiv

  38. [46]

    Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2024. https://doi.org/10.48550/ARXIV.2410.02694 HELMET: how to evaluate long-context language models effectively and thoroughly . CoRR, abs/2410.02694

    2024 doi arXiv

  39. [47]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Onta \ n \' o n, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. https://proceedings.neurips.cc/paper/2020/hash/c8512d142a2d849725f31a9a7a361ab9-Abstract.html Bi...

    2020

  40. [48]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/V1/P19-1472 Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, ...

    2019 doi

  41. [49]

    Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. 2024 a . Long context compression with activation beacon. arXiv preprint arXiv:2401.03462

    2024 arXiv

  42. [50]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024 b . https://doi.org/10.48550/ARXIV.2402.13718 \( \) bench: Extending long context evaluation beyond 100k tokens . CoRR, abs/2402.13718

    2024 doi arXiv

  43. [51]

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji - Rong Wen. 2024 c . https://doi.org/10.48550/ARXIV.2404.13501 A survey on the memory mechanism of large language model based agents . CoRR, abs/2404.13501

    2024 doi arXiv

  44. [52]

    Barrett, Zhangyang Wang, and Beidi Chen

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \' e , Clark W. Barrett, Zhangyang Wang, and Beidi Chen. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/6ceefa7b15572587b78ecfcebb2827f8-Abstract-C...

    2023

  45. [53]

    https://huggingface.co/datasets/hoskinson-center/proof-pile Proofpile: A pre-training dataset of mathematical texts

    Bartosz Piotrowski Zhangir Azerbayev, Edward Ayers. https://huggingface.co/datasets/hoskinson-center/proof-pile Proofpile: A pre-training dataset of mathematical texts

  46. [54]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian - Yun Nie, and Ji - Ro...

    2023 doi arXiv

  47. [55]

    Yujia Zhou, Zhicheng Dou, Huaying Yuan, and Zhengyi Ma. 2022. https://doi.org/10.1145/3485447.3511962 Socialformer: Social network inspired long document modeling for document ranking . In WWW '22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022 ,...

    2022

  48. [56]

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji - Rong Wen. 2023. https://doi.org/10.48550/ARXIV.2308.07107 Large language models for information retrieval: A survey . CoRR, abs/2308.07107

    2023 doi

  49. [57]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  50. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.