Pith. sign in

REVIEW 4 major objections 4 minor 300 references

Tokens that later context predicts well are redundant and safe to evict from a language model's key-value cache.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:53 UTC pith:VQWOWQYN

load-bearing objection A promising training-free eviction heuristic, but the 'counter-causal surprise' score leaks the token's own information through cached future states, so the paper's mechanistic story needs rework. the 4 major comments →

arxiv 2607.27600 v1 pith:VQWOWQYN submitted 2026-07-30 cs.LG

Back from the Future: Key-Value Cache Management by Counter-Causal Surprise

Stephen Gould , Anton van den Hengel This is my paper
classification cs.LG
keywords KV cachecounter-causal surpriseevictionLLM inferenceattentionmemory managementlong-contextdecoding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a token's predictability from its own future context is a better criterion for cache eviction than accumulated attention weight. The method, counter-causal surprise, runs a reverse-masked attention pass over the already-cached keys and values, scores each past token by how well later tokens predict it, and evicts low-scoring entries. The authors argue this avoids the self-reinforcing bias of attention-based eviction, which can drop rare but critical facts. On math, long-document, and long-conversation benchmarks the approach matches or beats prior eviction methods at the same cache size. A single-layer approximation cuts refresh cost seven to nine times.

Core claim

The authors propose that if the present context predicts a past token well, the model need not remember that token to predict the future. They implement this as counter-causal surprise, computed by an upper-triangular attention mask over cached keys and values, reusing the existing cache in-distribution with no training. Tokens with high surprise are retained; low-surprise tokens are evicted. The central empirical finding is that this criterion preserves generation accuracy at fixed cache budgets on MATH500, AIME thinking-mode, LongHealth, Qasper, and LoCoMo, and in particular protects rarely-attended facts, where attention-based eviction fails.

What carries the argument

The central object is the counter-causal surprise score s_i = 1 - P(x_i | x_{i+1:t}), approximated by running the model's forward pass with an upper-triangular attention mask over the cached key and value tensors (re-using stored K/V with original RoPE positions) and taking the logit of x_i. The score ranks which KV pairs are redundant: low surprise means later context already carries the information. The fast variant restricts the pass to the last layer, storing penultimate activations, to cut refresh cost to O(n^2).

Load-bearing premise

The load-bearing premise is that if later tokens can predict a past token well, then that past token carries no information the model will need later — but a token can be locally predictable and still be the only place a fact is stored for exact retrieval later.

What would settle it

Construct a synthetic context with a unique but highly predictable fact (e.g., a name appearing once in a predictable phrase) required to answer a later question, then run the method at small cache size: if the fact token is evicted and the answer degrades compared to attention-based eviction, the redundancy premise fails for exact retrieval.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A training-free scoring pass over the existing cache can serve as a principled eviction policy for any autoregressive transformer with a KV cache.
  • Because surprise is computed from future context rather than accumulated attention, it does not suffer from the self-reinforcing retention bias of attention-based heavy-hitter methods.
  • The last token in the cache always receives maximum surprise and is always retained, giving a built-in recency bias.
  • The fast single-layer approximation provides most of the accuracy gain at 7-9x lower refresh latency, making the approach practical for long-context decode.
  • For decode-heavy settings like thinking-mode reasoning, counter-causal eviction preserves reasoning coherence better than sliding window, importance, or H2O at high eviction pressure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to combine counter-causal scoring with phrase- or sentence-level eviction, which the paper itself flags, to protect tokens that are locally predictable but globally irreplaceable (e.g., names or numbers).
  • The scoring could be applied as a post-hoc compression of already-generated prompts without re-running the model, since it reuses cached keys and values rather than requiring a fresh forward pass.
  • The counter-causal logit might also serve as a signal for cache compression (merging redundant values) rather than only hard eviction.
  • The method's weakness on exact retrieval suggests a hybrid policy that adds a retrieval-aware bonus for entity tokens, testable on fact-retrieval benchmarks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a training-free KV cache eviction scheme based on a counter-causal surprise score. For each cached token, the score is computed by running the model with an upper-triangular attention mask over the already-cached keys and values, so that each position attends only to later positions. Tokens with high surprise (poorly predicted by their future context) are retained; tokens with low surprise are evicted. A faster variant restricts the scoring pass to the last transformer layer. The method is evaluated on MATH500, AIME, LongHealth, Qasper, and LoCoMo with Qwen2.5-3B/7B/14B and LLaMA-3.1-8B, comparing against sliding-window, an attention-importance baseline, and a simplified H2O baseline. The authors report competitive or improved accuracy at fixed cache sizes, with a 7--9x refresh speedup for the fast variant. The central claim is that counter-causal surprise identifies tokens whose information is already contained in later context, avoiding the self-reinforcing attention bias of H2O/TOVA-style methods.

Significance. If the central claim were fully supported, the paper would contribute a simple, model-agnostic, training-free eviction signal that is fundamentally different from attention-based scores, with potential value in long-context and long-decoding settings. The idea of using the model's own predictive ability to estimate redundancy is appealing, and the authors provide reference code and report a careful efficiency analysis of the refresh overhead. However, the empirical validation is weakened by two factors: the scoring procedure appears to suffer from causal self-leakage that undermines the counter-causal interpretation, and the comparisons are against simplified or self-implemented baselines rather than the actual cited state-of-the-art methods. These issues are load-bearing for the paper's main claim, so the result, while potentially useful as a heuristic, is not yet established at the level claimed.

major comments (4)
  1. [§3.1, Eq. (5)] The score is not actually P(x_i | x_{i+1:t}) as claimed. The cached K/V for positions j>i were computed by the standard causal forward pass, so the hidden states at j already encode information about x_i through attention over earlier positions. When position i attends to those cached future representations, the predictor has direct access to x_i itself. Thus the logit does not measure whether the remaining cache can 'compensate for the absence of xi' — the future representations were constructed with xi present. The Limitations paragraph admits an approximation but does not identify this self-leakage or quantify its effect. Please either re-compute future hidden states with x_i masked out (or otherwise characterize the leakage) and compare rankings, or substantially revise the interpretation of the score and the claims that depend on it.
  2. [§4, Tables 2–3 and Fig. 5] The experimental comparison is against simplified versions of the cited methods. 'Importance' is described as 'a simplified version of the heavy-hitter oracle' and 'similar in spirit to TOVA'; 'Heavy-hitter' is implemented using 'the most recent keys as a proxy for queries'. The actual H2O and TOVA algorithms, and the layer-adaptive methods PyramidKV and Ada-KV, are not run or compared with published numbers. The abstract and conclusion claim 'competitive or improved performance over other state-of-the-art methods', but the evidence supports only a comparison with the authors' simplified baselines. Please compare against reference implementations of at least H2O and TOVA, and include PyramidKV/Ada-KV or justify their exclusion with direct experiments rather than an argument about shared attention bias.
  3. [§4.2, Table 2; §4.3, Table 3] All reported accuracies come from a single greedy rollout with no error bars or multiple seeds. Several differences are small (e.g., 0.744 vs 0.762 on Qwen2.5-7B in Table 2; 0.367 vs 0.333 in Table 3), so without variance estimates the reported ordering may not be statistically meaningful. Please provide repeated-run statistics or seed variance for at least one benchmark per setting, and apply the same to Figure 5.
  4. [Limitations] The manuscript itself acknowledges that 'some individual tokens may be easily predicted from nearby context causing counter-causal surprise to underestimate their importance on exact retrieval tasks and evict them from the cache.' This is a load-bearing limitation for the central claim, because the method's rationale is that low surprise implies redundancy for future prediction. LoCoMo, one of the key benchmarks, requires exact-phrase retrieval from long conversations, and the paper's own discussion of H2O failures shows that exact-retrieval failures are the main source of degradation. Please add an explicit exact-retrieval stress test or report the failure mode on LoCoMo, and qualify the central claim accordingly.
minor comments (4)
  1. [§3.1 / Algorithm] The text states that the last token is assigned maximum surprise and always retained, but the pseudocode in DOMEMREFRESH does not implement this exception explicitly (it simply takes the top-J of all scores). Clarify which behavior is used in experiments.
  2. [§3.1 / Algorithm] The pseudocode uses the probability notation '1 − P(X_t | X_{t+1:J+h})' while Section 3.1 says the implementation uses the logit directly. Make the pseudocode consistent with the implementation.
  3. [Figure 5] The figure appears multiple times with duplicated subfigures in the manuscript text. Please keep a single copy of the composite figure and refer to it in the text.
  4. [§4.3] The AIME dataset is cited to a 2026 arXiv paper, which may not be familiar. Provide the standard dataset source or a URL for reproducibility.

Circularity Check

0 steps flagged

No circularity found: the counter-causal surprise score is defined from the model's own cached representations and evaluated on external benchmarks; the core insight is a heuristic assumption, not a derived result that reduces to its inputs.

full rationale

The paper's eviction method is a heuristic whose score is defined directly from model quantities (Eq. 5: s_i = 1 - P(x_i | x_{i+1:t}), approximated by a counter-causal attention pass reusing cached K/V). This score is then used to select cache entries, and the method's quality is measured against external benchmarks (MATH500, AIME, LongHealth, Qasper, LoCoMo) relative to a full-cache baseline. No parameter is fitted to the evaluation data, and the eviction decision is not asserted to be correct by construction: the paper explicitly frames 'if the present is good at predicting the past, then we do not need to remember the past' as a motivating insight rather than a theorem, and the Limitations section concedes both that the counter-causal score is only an approximation and that predictable tokens may still be needed for exact retrieval. These are soundness/approximation concerns, not circularity. The paper does not rename an existing result, does not import a load-bearing uniqueness theorem from self-citations, and does not reduce any prediction to a fitted input. The main empirical claim is tested against external baselines and thus is not forced by the method's definitions.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

No new physical entities or fitted constants. The paper introduces an algorithmic score (counter-causal surprise) and relies on two domain assumptions: the redundancy implication and the reverse-pass approximation. Cache size and chunk size are hand-set hyperparameters, not fitted to data.

free parameters (3)
  • cache size J = 512 (MATH500/AIME), 4096 (AIME), 4000-24000 sweeps (LongHealth/Qasper/LoCoMo)
    Memory budget; chosen per task, not learned. Performance depends on it.
  • chunk size h = 256 (MATH500), 2048 (AIME), 25% of cache size elsewhere
    Refresh interval; hand-set hyperparameter. The comparison changes with refresh frequency.
  • frozen system-prompt sink slots = small number, not specified exactly
    Tokens/slots frozen from eviction, used for all strategies; count is not given.
axioms (3)
  • domain assumption If a past token is predictable from future tokens, its KV pair is redundant for future prediction.
    Core insight in Section 3 / footnote 3; not proven; load-bearing premise.
  • domain assumption Reusing causally computed cached K/V with an upper-triangular mask approximates P(x_i | x_{i+1:t}).
    Section 3.1 and Limitations; the approximation error is unquantified.
  • standard math Cached RoPE keys/values can be combined with freshly computed queries at original positions to produce valid counter-causal attention scores.
    Depends on RoPE properties; likely standard but not formally verified in the paper.

pith-pipeline@v1.3.0-daily-deepseek · 19445 in / 16047 out tokens · 175975 ms · 2026-08-01T04:53:00.524502+00:00 · methodology

0 comments
read the original abstract

Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years. Computational demands of large language models (LLMs) and their multi-modal variants during output generation can be partially alleviated by caching previous key and value calculations needed by subsequent scaled dot-product attention operations. However, this leads to another problem: the size of the resulting KV cache grows linearly with context length and quickly consumes all available GPU memory when either the prompt or the generated output are long. KV cache management periodically prunes entries from the cache thereby reducing its memory footprint while attempting to retain sufficient information for accurate generation. A by-product is faster inference speed. We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant and their associated keys and values can be removed from the cache. To score entries for eviction we run the model on the tokens in their original order, reusing the key and value representations already stored in the KV cache, and applying a counter-causal attention mask so that each position attends only to its future context. This is in-distribution, tied directly to the actual cache contents, and requires no additional training. To further reduce cost, we additionally propose a fast single-layer approximation that restricts the counter-causal pass to the last transformer layer, achieving a significant speedup per refresh cycle at marginal accuracy cost. We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods. Reference code is available at https://github.com/metacognitionai/counter_causal.

Figures

Figures reproduced from arXiv: 2607.27600 by Anton van den Hengel, Stephen Gould.

Figure 1
Figure 1. Figure 1: General framework KV cache management with inference and memory refresh cycles. Our [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: MATH500 accuracy by model and eviction strategy (same data as Table 2). [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Histogram of generated output sequence length by number of characters (left) and number [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of the memory augmented key-value cache for different strategies on [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Experimental results on three different long context tasks as a function of cache size [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

300 extracted references · 13 linked inside Pith

  1. [1]

    Chamin Hewa Koneputugodage and Yizhak Ben-Shabat and Dylan Campbell and Stephen Gould , title =

  2. [2]

    Stephen Gould , title =

  3. [3]

    Molloy and Stephen Gould , title =

    Ming Xu and Timothy L. Molloy and Stephen Gould , title =

  4. [4]

    First AAAI Workshop on Optimal Transport and Structured Data Modeling (

    Stephen Gould and Dylan Campbell and Yizhak Ben-Shabat and Chamin Hewa Koneputugodage and Zhiwei Xu , title =. First AAAI Workshop on Optimal Transport and Structured Data Modeling (

  5. [5]

    Yizhak Ben-Shabat and Chamin Hewa Koneputugodage and Stephen Gould , title =

  6. [6]

    Stephen Gould and Richard Hartley and Dylan Campbell , title =

  7. [7]

    Dylan Campbell and Liu Liu and Stephen Gould , title =

  8. [8]

    arXiv:1909.04866 , institution =

    Stephen Gould and Richard Hartley and Dylan Campbell , title =. arXiv:1909.04866 , institution =

    Pith/arXiv arXiv 1909

  9. [9]

    Visual Permutation Learning , journal = PAMI, year =

    Rodrigo. Visual Permutation Learning , journal = PAMI, year =

  10. [10]

    Peter Anderson and Stephen Gould and Mark Johnson , title =

  11. [11]

    Anoop Cherian and Stephen Gould , title =

  12. [12]

    Anoop Cherian and Suvrit Sra and Stephen Gould and Richard Hartley , title =

  13. [13]

    Peter Anderson and Xiaodong He and Chris Buehler and Damien Teney and Mark Johnson and Stephen Gould and Lei Zhang , title =

  14. [14]

    Peter Anderson and Qi Wu and Damien Teney and Jake Bruce and Mark Johnson and Niko Sunderhauf and Ian Reid and Stephen Gould and Anton van den Hengel , title =

  15. [15]

    Jue Wang and Anoop Cherian and Fatih Porikli and Stephen Gould , title =

  16. [16]

    Neural Algebra of Classifiers , booktitle = WACV, year =

    Rodrigo. Neural Algebra of Classifiers , booktitle = WACV, year =

  17. [17]

    Peter Anderson and Basura Fernando and Mark Johnson and Stephen Gould , title =

  18. [18]

    Alvarez and Stephen Gould , title =

    Fatemeh Sadat Saleh and Mohammad Sadegh Aliakbarian and Mathieu Salzmann and Lars Petersson and Jose M. Alvarez and Stephen Gould , title =

  19. [19]

    Basura Fernando and Stephen Gould , title =

  20. [20]

    Anoop Cherian and Basura Fernando and Mehrtash Harandi and Stephen Gould , title =

  21. [21]

    Visual Permutation Learning , booktitle = CVPR, year =

    Rodrigo. Visual Permutation Learning , booktitle = CVPR, year =

  22. [22]

    Basura Fernando and Hakan Bilen and Efstratios Gavves and Stephen Gould , title =

  23. [23]

    Alvarez , title =

    Fatemehsadat Saleh and Mohammad Sadegh Ali Akbarian and Mathieu Salzmann and Lars Petersson and Stephen Gould and Jose M. Alvarez , title =

  24. [24]

    On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization , eprint =

    Stephen Gould and Basura Fernando and Anoop Cherian and Peter Anderson and Rodrigo. On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization , eprint =

  25. [25]

    Bioimage Computing at ECCV , year =

    Aisha Khan and Stephen Gould and Mathieu Salzmann , title =. Bioimage Computing at ECCV , year =

  26. [26]

    Basura Fernando and Peter Anderson and Marcus Hutter and Stephen Gould , title =

  27. [27]

    Hakan Bilen and Basura Fernando and Stratis Gavves and Andrea Vedaldi and Stephen Gould , title =

  28. [28]

    Communications of the ACM (CACM) , pages =

    Stephen Gould and Xuming He , title =. Communications of the ACM (CACM) , pages =

  29. [29]

    Stephen Gould and Jiecheng Zhao and Xuming He and Yuhang Zhang , title =

  30. [30]

    Stephen Gould and Yuhang Zhang , title =

  31. [31]

    Kyoungup Park and Stephen Gould , title =

  32. [32]

    Digital Image Computing: Techniques and Applications (

    Paul Rivera and Stephen Gould , title =. Digital Image Computing: Techniques and Applications (

  33. [33]

    Beyang Liu and Stephen Gould and Daphne Koller , title =

  34. [34]

    Stephen Gould and Tianshi Gao and Daphne Koller , title =

  35. [35]

    Stephen Gould and Rick Fulton and Daphne Koller , title =

  36. [36]

    Stephen Gould and Fernando Amat and Daphne Koller , title =

  37. [37]

    Le and Ashley Wellman and Andrew Y

    Morgan Quigley and Siddharth Batra and Stephen Gould and Ellen Klingbeil and Quoc V. Le and Ashley Wellman and Andrew Y. Ng , title =

  38. [38]

    Geremy Heitz and Stephen Gould and Ashutosh Saxena and Daphne Koller , title =

  39. [39]

    Stephen Gould and Jim Rodgers and David Cohen and Gal Elidan and Daphne Koller , title =

  40. [40]

    Gal Elidan and Stephen Gould , title =

  41. [41]

    2008 , volume =

    Gal Elidan and Stephen Gould , title =. 2008 , volume =

    2008

  42. [42]

    Duchi and Stephen Gould and Daphne Koller , title =

    John C. Duchi and Stephen Gould and Daphne Koller , title =

  43. [43]

    Ng and Daphne Koller , title =

    Stephen Gould and Paul Baumstarck and Morgan Quigley and Andrew Y. Ng and Daphne Koller , title =. ECCV Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications (M2SFA2) , year =

  44. [44]

    Bradski and Paul Baumstarck and Sukwon Chung and Andrew Y

    Stephen Gould and Joakim Arfvidsson and Adrian Kaehler and Benjamin Sapp and Marius Messner and Gary R. Bradski and Paul Baumstarck and Sukwon Chung and Andrew Y. Ng , title =

  45. [45]

    Beyond Benchmarks:

    Jasper Dekoninck and Nikola Jovanovic and Tim Gehrunger and Kari Rognvaldsson and Ivo Petrov and Chenhao Sun and Martin Vechev , journal=. Beyond Benchmarks:

  46. [46]

    Vision Transformers Need Registers , author=

  47. [47]

    arXiv 2412.15115 , year=

    Qwen2.5 Technical Report , author=. arXiv 2412.15115 , year=

    Pith/arXiv arXiv

  48. [48]

    arXiv 2407.21783 , year=

    The Llama 3 Herd of Models , author=. arXiv 2407.21783 , year=

    Pith/arXiv arXiv

  49. [49]

    Guangxuan Xiao and Yuandong Tian and Beidi Chen and Song Han and Mike Lewis , title =

  50. [50]

    Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs , author=

  51. [51]

    Li, Yuhong and Huang, Yingbing and Yang, Bowen and Venkitesh, Bharat and Locatelli, Acyr and Ye, Hanchen and Cai, Tianle and Lewis, Patrick and Chen, Deming , journal=

  52. [52]

    arXiv 2004.05150 , year=

    Longformer: The Long-Document Transformer , author=. arXiv 2004.05150 , year=

    Pith/arXiv arXiv 2004

  53. [53]

    and Ermon, Stefano and Rudra, Atri and R

    Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R. Flash

  54. [54]

    2023 , booktitle =

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph and Zhang, Hao and Stoica, Ion , title =. 2023 , booktitle =

    2023

  55. [55]

    and Li, Dongsheng and Lin, Chin-Yew and Yang, Yuqing and Qiu, Lili , title =

    Jiang, Huiqiang and Li, Yucheng and Zhang, Chengruidong and Wu, Qianhui and Luo, Xufang and Ahn, Surin and Han, Zhenhua and Abdi, Amir H. and Li, Dongsheng and Lin, Chin-Yew and Yang, Yuqing and Qiu, Lili , title =

  56. [56]

    2019 , journal=

    Generating Long Sequences with Sparse Transformers , author=. 2019 , journal=

    2019

  57. [57]

    Low-Rank Approximation for Sparse Attention in Multi-Modal

    Lin Song and Yukang Chen and Shuai Yang and Xiaohan Ding and Yixiao Ge and Ying-Cong Chen and Ying Shan , booktitle=CVPR, year=. Low-Rank Approximation for Sparse Attention in Multi-Modal

  58. [58]

    and Gardner, Matt

    Dasigi, Pradeep and Lo, Kyle and Beltagy, Iz and Cohan, Arman and Smith, Noah A. and Gardner, Matt. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). 2021

    2021

  59. [59]

    Measuring Mathematical Problem Solving With the

    Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt , journal=NIPS, year=. Measuring Mathematical Problem Solving With the

  60. [60]

    Yushi Bai and Shangqing Tu and Jiajie Zhang and Hao Peng and Xiaozhi Wang and Xin Lv and Shulin Cao and Jiazheng Xu and Lei Hou and Yuxiao Dong and Jie Tang and Juanzi Li , journal=

  61. [61]

    arXiv preprint arXiv:2401.14490 , year=

    Adams, Lisa and Busch, Felix and Han, Tianyu and Excoffier, Jean-Baptiste and Ortala, Matthieu and L. arXiv preprint arXiv:2401.14490 , year=

    Pith/arXiv arXiv

  62. [62]

    arxiv , year =

    Maharana, Adyasha and Lee, Dong-Ho and Tulyakov, Sergey and Bansal, Mohit and Barbieri, Francesco and Fang, Yuwei , title =. arxiv , year =

  63. [63]

    arXiv:2506.06266 , year =

    Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study , author =. arXiv:2506.06266 , year =

    Pith/arXiv arXiv

  64. [64]

    An Evolved Universal Transformer Memory , author=

  65. [65]

    H _2 O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models , author=

  66. [66]

    Amir Zandieh and Majid Daliri and Majid Hadian and Vahab Mirrokni , booktitle = ICLR, year =

  67. [67]

    T oken S kip: Controllable Chain-of-Thought Compression in LLM s

    Xia, Heming and Leong, Chak Tou and Wang, Wenjie and Li, Yongqi and Li, Wenjie. T oken S kip: Controllable Chain-of-Thought Compression in LLM s. 2025

    2025

  68. [68]

    Beyond Speedup: Utilizing KV Cache for Sampling and Reasoning , author =

  69. [69]

    Augmenting Self-Attention with Persistent Memory , author =

  70. [70]

    arXiv:2501.00663 , year =

    Titans: Learning to Memorize at Test Time , author =. arXiv:2501.00663 , year =

    Pith/arXiv arXiv

  71. [71]

    Joshua Ainslie and James Lee-Thorp and Michiel de Jong and Yury Zemlyanskiy and Federico Lebrón and Sumit Sanghai , booktitle = EMNLP, year=

  72. [72]

    Hooper, Coleman and Kim, Sehoon and Mohammadzadeh, Hiva and Mahoney, Michael W and Shao, Yakun Sophia and Keutzer, Kurt and Gholami, Amir , journal=

  73. [73]

    Language Models are Few-Shot Learners , year =

    Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winte...

  74. [74]

    Compressing Context to Enhance Inference Efficiency of Large Language Models , author=

  75. [75]

    Extending Context Window of Large Language Models via Semantic Compression

    Fei, Weizhi and Niu, Xueyan and Zhou, Pingyi and Hou, Lu and Bai, Bo and Deng, Lei and Han, Wei. Extending Context Window of Large Language Models via Semantic Compression. 2024

    2024

  76. [76]

    2012.06678 , archivePrefix=

    Xin Huang and Ashish Khetan and Milan Cvitkovic and Zohar Karnin , year=. 2012.06678 , archivePrefix=

    Pith/arXiv arXiv 2012

  77. [77]

    Revisiting Deep Learning Models for Tabular Data , author=

  78. [78]

    arXiv preprint arXiv:1807.03748 , year =

    Representation learning with contrastive predictive coding , author =. arXiv preprint arXiv:1807.03748 , year =

    Pith/arXiv arXiv

  79. [79]

    Exploring simple siamese representation learning , author =

  80. [80]

    Bootstrap your own latent---a new approach to self-supervised learning , author =

Showing first 80 references.