Pith. sign in

REVIEW 3 major objections 5 minor 100 references

ARMT extends LLM context beyond original limits with constant memory and 30% fewer FLOPs while matching in-window performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 04:22 UTC pith:I5I3UHUE

load-bearing objection Solid systems extension of ARMT to 1B-scale domain SFT with real OOD length retention and constant memory; the abstract oversells the FLOP and "no degradation" packaging, but the empirical core holds. the 3 major comments →

arxiv 2607.11614 v1 pith:I5I3UHUE submitted 2026-07-13 cs.CL cs.AI

Extending LLM Context via Associative Recurrent Memory

Gleb Kuzmin , Ivan Rodkin , Aydar Bulatov , Yuri Kuratov , Lyudmila Rvanova , Mikhail Katkov , Ilia Sochenkov , Misha Tsodyks
show 3 more authors
Timothy Baldwin Mikhail Burtsev Artem Shelmanov
This is my paper
classification cs.CL cs.AI
keywords long-context LLMsassociative recurrent memoryARMTcontext extensionconstant memorycurriculum learningdomain-specific fine-tuningsegment-wise attention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard transformers hit a wall on long documents because self-attention grows quadratically in length and memory grows linearly. This paper shows that wrapping a pre-trained LLM in the Associative Recurrent Memory Transformer (ARMT) lets the model process sequences far past its original context window without quality loss relative to the same model run inside that window. The method splits input into short fixed segments that keep full local attention, then uses a small set of associative memory layers to pass the important facts from segment to segment, so global memory stays constant and total FLOPs drop by roughly thirty percent. The authors supply two realistic domain datasets (code type prediction and long-document QA) plus a practical training recipe—continued pre-training, synthetic long-context data, curriculum lengthening, and optional pruning of associative layers—that makes the approach work on models as small as a few hundred million parameters. The result is a way for local, privacy-sensitive models to handle tens of thousands of tokens without sending data to a large remote service.

Core claim

When a pre-trained LLM is augmented with ARMT and trained with the authors’ recipe, the resulting model processes inputs well beyond its original context limit without degrading performance relative to the same model run inside that limit, generalizes better to out-of-distribution lengths, and requires approximately 30 percent fewer FLOPs while matching baseline quality inside the original window.

What carries the argument

The Associative Recurrent Memory Transformer (ARMT): a wrapper that divides the input into fixed-length segments, retains full self-attention inside each segment, and inserts a small number of associative blocks that compress each segment into key–value memory and inject the recalled associations into the next segment, thereby giving linear scaling and constant memory.

Load-bearing premise

A fixed 1,024-token segment size together with only a few carefully chosen associative layers is enough to carry the task-critical information across dozens of segments on real domain data, so that piecewise attention plus recurrent memory can fully replace full self-attention.

What would settle it

On the same ManyTypes-long or GovReport-long test sets, an ARMT model whose associative layers have been removed (or whose segment size is increased until memory no longer spans the critical facts) shows a sharp drop on the 32k–65k “Long-OOD” band while the full-attention baseline does not.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Small local LLMs can be given 32k–65k context on domain tasks without quadratic memory growth or remote API calls.
  • Only about 20 percent of layers need associative memory, cutting trainable parameters and fine-tuning time by roughly 30 percent.
  • Curriculum lengthening plus synthetic long-context data is sufficient to initialize the memory mechanism even when real long documents are scarce.
  • The same recipe works on both code-type prediction and long-document QA, suggesting domain-specific long-context fine-tuning is broadly feasible.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same segment-plus-associative pattern could be dropped into other hybrid architectures (sliding-window plus recurrent state) that already trade full attention for efficiency.
  • If the critical layers turn out to be architecture-dependent rather than task-dependent, a single pre-selection rule could be shipped with each base model family.
  • Privacy-sensitive deployments that today refuse long-context APIs may become viable once constant-memory ARMT fine-tunes are available for the 1B-parameter class.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a practical recipe for extending the context of small pretrained LLMs (Gemma-3-1B-IT, SmolLM-2-360M-IT) by wrapping them with Associative Recurrent Memory Transformer (ARMT) layers: fixed-size segments with full local attention plus a recurrent associative memory that propagates key–value associations across segments. Contributions include two domain-specific long-context datasets (ManyTypes-long for code type prediction; GovReport-long / GR-100+ for document QA, with synthetic augmentation), a training pipeline (continued LM pre-training on FineWeb-Edu, synthetic long-context QA, curriculum over 2/4/8 segments, and optional pruning or pre-selection of associative layers), and experiments showing that ARMT models maintain competitive in-window accuracy, improve Long-OOD generalization beyond the base model’s native limit (32k/8k), use constant GPU memory, and are claimed to require ~30% fewer FLOPs. Supporting evidence includes multi-length tables, ablations on pre-training, curriculum, synthetic data, memory size, RMT vs ARMT, and comparisons to Mamba/Mamba-2/DeltaNet/xLSTM/YaRN plus BABILong and ContractNLI.

Significance. If the claims hold under fair accounting, the work is a useful systems contribution for privacy-sensitive, domain-specific deployments of small local models: constant memory, linear scaling, and a concrete fine-tuning recipe that reuses existing LLMs rather than training recurrent models from scratch. Strengths include extensive length-binned evaluation (Tables 1–2, 8–11), layer-importance and pre-selection ablations (Tables 3–4, 12–20), synthetic-data scaling (Fig. 5), and external-benchmark checks (BABILong, ContractNLI). The universal pre-selection of ~20% of layers (e.g., 7, 13, 14, 19, 25) that matches full ARMT while cutting trainable parameters and training time by ~30% is a practical finding. The paper is empirical and reproducible in spirit; the FLOP derivation in §A.1 is transparent even if approximate.

major comments (3)
  1. [Abstract; §5.3; Tables 1–2] Abstract and §5.2–5.3 package three results as (i) no degradation vs in-limit baselines, (ii) better OOD generalization, and (iii) ~30% fewer FLOPs while preserving in-window performance. Claim (i) is overstated relative to the tables: on GR-100+ (Table 2) full ARMT in-domain ROUGE-L is 0.299 vs base 0.351; on MT (Table 1) EM is 0.749 vs 0.797. The accurate story is “comparable overall + better Long-OOD,” which should replace the stronger packaging in the abstract and conclusion.
  2. [§5.2; §A.1; Fig. 2] The 30% FLOP claim (§5.2, §A.1) rests on an incomplete accounting: local-attention cost, memory-token overhead, and the sequential segment schedule are ignored, yielding FARMT/FGemma≈0.67 for T=32k, S=1k. Empirically Fig. 2b shows ARMT slower than the base model at matched batch size 8; wall-clock improves only after raising ARMT batch size to 32 (enabled by constant memory). A same-batch, same-hardware FLOP/throughput comparison that includes the associative-block cost and sequential loop is needed before the efficiency claim can support the abstract’s third bullet.
  3. [§5.1; Fig. 3] The experimental setup (§5.1) deliberately shrinks the native attention window to 1,024 tokens so that long-context burden falls on recurrent memory (32 segments for 32k). This is a valid stress test of ARMT, but it is not a head-to-head “context extension of the original LLM” under the base model’s full attention budget. The paper should state more clearly that the comparison is piecewise attention + memory vs full attention at the base’s original window, and report at least one run where ARMT segment size is closer to the base window so readers can separate memory benefit from forced truncation of the baseline.
minor comments (5)
  1. [§3; Table 4] Clarify in §3 and Table 4 whether “pre-selected layers” are chosen once from the full-model ablation or fixed a priori by the universal recipe; the text mixes both narratives.
  2. [Fig. 2; §5.2] Fig. 2 caption and §5.2 should report absolute tokens/s and hardware (H100) so the batch-size-32 advantage is interpretable.
  3. [§4; Appendix B] Dataset construction (question before context; paragraph stacking) is sensible but should note possible distribution shift vs natural long documents; a short qualitative error analysis on Long-OOD failures would help.
  4. [Abstract; Appendix A] Typos and notation: “inputswell”, “exhibitsupe-rior”, “need 30% less FLOPs” in abstract; associative-matrix update equations in Appendix A would benefit from a short prose walk-through of β, γ, and ϕ.
  5. [Limitations; §6] Limitations correctly note the 1B-scale restriction; a one-sentence discussion of whether the pre-selection recipe is expected to transfer to larger models would strengthen the conclusion.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper whose performance and FLOP claims are measured or derived independently of the inputs they are claimed to support.

full rationale

The paper is an empirical architecture-and-training study. Its three headline results (long-context extension without in-window degradation, superior OOD length generalization, and ~30% fewer FLOPs) are obtained from held-out test splits of newly constructed domain datasets (MT, GR-100+), external benchmarks (BABILong, ContractNLI), and a standard attention/FFN FLOP accounting given in Appendix A.1. The FLOP ratio is obtained by substituting model constants (Ng, H, dh, N, d, T=32768, S=1024) into the usual quadratic-attention plus linear-FFN expressions; it is not fitted to the accuracy numbers. Self-citations (Rodkin et al. 2024 for the ARMT associative block, Bulatov et al. for RMT, etc.) supply the architectural definition and prior small-scale results; they do not supply the measured long-context scores or the FLOP derivation. Curriculum learning, synthetic-data generation, and layer pre-selection are training heuristics whose success is evaluated on held-out data, not tautological by construction. No uniqueness theorem, self-definitional identity, or fitted-parameter-as-prediction appears. Circularity burden is therefore zero.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The central claims rest on the ARMT associative update equations imported from prior work, standard transformer/LoRA training assumptions, hand-chosen architectural knobs (segment length, memory tokens, which layers receive associative blocks), and the construction of synthetic long contexts that preserve short-passage QA labels. No new physical entity is postulated; free parameters are engineering choices that affect measured accuracy and the FLOP approximation.

free parameters (6)
  • segment_size_S
    Fixed at 1,024 tokens; defines the local full-attention window and the T/S FLOP reduction factor used for the 30% claim.
  • num_memory_tokens
    16 for non-pretrained ARMT, 32 after continued pretraining; chosen by ablation (Tables 34–35), not derived.
  • associative_layer_subset
    Pre-selected layers (e.g., 7,13,14,19,25 for Gemma-3-1B) or top-k by post-hoc importance; heuristic recipe, not theoretically fixed.
  • curriculum_segment_schedule
    2→4→8 segments with annealed LR; schedule and step counts (Table 33) are hand-tuned.
  • synthetic_to_real_ratio_GR100plus
    Best reported ratio ~5.5 synthetic:real for GR-100+ (Fig. 5); selected by scaling sweep.
  • LoRA_rank_alpha
    rank=64, α=128 on all linear layers; standard but free engineering choice affecting capacity.
axioms (5)
  • domain assumption ARMT associative block equations (memory extraction, consolidation into A^l_s, query association) correctly implement capacious segment-level long-term memory as defined in Rodkin et al. 2024.
    Imported wholesale in §2 and Appendix A; not re-derived here.
  • domain assumption Continued unsupervised LM pretraining on multi-segment sequences initializes associative parameters so that later SFT can exploit cross-segment propagation.
    Stated as hypothesis in §3; supported empirically but not proven generally.
  • ad hoc to paper Concatenating short passages with their QA labels yields valid long-context supervision that tests memory rather than parametric knowledge (question placed before context).
    Core of synthetic GR and curriculum construction in §3–4.
  • domain assumption Standard transformer attention/FFN FLOP accounting (4 N_g H d_h T^2 + 30 N T d^2) with M≪S is accurate enough to claim ~30% total FLOP reduction.
    §A.1 approximation; local attention and memory-token overhead neglected.
  • domain assumption LoRA fine-tuning of backbone plus associative parameters is sufficient to adapt ARMT without full-model SFT.
    Training setup §5.1; common practice but load-bearing for reported scores.
invented entities (3)
  • ManyTypes-long (MT) dataset no independent evidence
    purpose: Domain long-context type-prediction benchmark built by stacking repository scripts up to 64k tokens.
    New constructed resource; independent of ARMT math but required for the reported EM results.
  • GovReport-long / GR-100+ dataset no independent evidence
    purpose: Long-document QA set with synthetic augmentation for curriculum bins up to 64k.
    New constructed resource with synthetic ratio chosen by authors.
  • Universal associative-layer pre-selection recipe no independent evidence
    purpose: Choose ~20% of layers (middle, final, quarter boundaries) before training to match full-ARMT quality at lower cost.
    Heuristic introduced in §5.3; validated only on the two backbones and two tasks here.

pith-pipeline@v1.1.0-grok45 · 51999 in / 3874 out tokens · 38908 ms · 2026-07-14T04:22:32.202344+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Extending LLM Context via Associative Recurrent Memory." pith.science (2026). https://pith.science/paper/I5I3UHUE

@misc{pith2026260711614,
  author       = {Pith},
  title        = {Pith review of: Extending LLM Context via Associative Recurrent Memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I5I3UHUE}},
  note         = {Machine review of arXiv:2607.11614}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Extending the context length of large language models (LLMs) is critical for many real-world applications, yet standard transformers remain constrained by quadratic compute and linear memory scaling. In this work, we investigate the Associative Recurrent Memory Transformer (ARMT) as a practical approach for enabling long-context processing in LLMs, constant memory scaling, and better efficiency. We make three main contributions. First, we construct two domain-specific long-context datasets designed to evaluate realistic workloads, focusing on narrow-domain fine-tuning scenarios. Second, we propose a comprehensive training recipe for ARMT-based context extension, combining continued pre-training, synthetic long-context data generation, curriculum learning, and selective integration of associative memory into chosen model layers. Third, we present an extensive experimental study demonstrating that ARMT-augmented models: (i) process inputs well beyond their original context limits without degrading performance relative to in-limit baselines; (ii) generalize more effectively to out-of-distribution context lengths; and (iii) need 30% less FLOPs while preserving baseline performance within the original context window.

Figures

Figures reproduced from arXiv: 2607.11614 by Artem Shelmanov, Aydar Bulatov, Gleb Kuzmin, Ilia Sochenkov, Ivan Rodkin, Lyudmila Rvanova, Mikhail Burtsev, Mikhail Katkov, Misha Tsodyks, Timothy Baldwin, Yuri Kuratov.

Figure 1
Figure 1. Figure 1: Base LLM architecture (left) and ARMT architecture (right). ARMT divides the input text into segments [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: ARMT performance in terms of inference time and memory usage. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Main results. (1) Gemma-3-1B-IT (Base) with full attention and Mamba-2 trained with a context length of 8,192 show a sharp performance drop beyond 32k tokens. (2) ARMT model with Gemma-3-1B-IT backbone (ARMT) maintains stable performance across all evaluated context lengths and outperforms other models in the long out-of-distribution (Long-OOD) regimen (>32k). (3) ARMT with continued pre-training followed … view at source ↗
Figure 4
Figure 4. Figure 4: ARMT-augmented Gemma-3-1B-IT pre￾training on 19B tokens from the FineWeb-Edu dataset. Pre-training was performed with 8,192-token sequences, divided into 8 segments of 1024 tokens each. We ad￾ditionally report the final checkpoint loss on the train subset, averaged over the first 1, 2, 4, and 8 segments. D.2 ARMT Fine-Tuning After Pre-Training After ARMT pre-training, we also conducted fine￾tuning using th… view at source ↗
Figure 5
Figure 5. Figure 5: Scaling synthetic data for the GR dataset [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Main results with SmolLM-2-360M-IT model. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

100 extracted references · 18 linked inside Pith

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    doi:10.1145/322234.322243 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [8]

    arXiv preprint arXiv:2407.04841 , year=

    Associative recurrent memory transformer , author=. arXiv preprint arXiv:2407.04841 , year=

    Pith/arXiv arXiv

  9. [9]

    Kuratov, Yury and Bulatov, Aydar and Anokhin, Petr and Rodkin, Ivan and Sorokin, Dmitry and Sorokin, Artyom and Burtsev, Mikhail , journal=

  10. [10]

    HIBRIDS : Attention with Hierarchical Biases for Structure-aware Long Document Summarization

    Cao, Shuyang and Wang, Lu. HIBRIDS : Attention with Hierarchical Biases for Structure-aware Long Document Summarization. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022. doi:10.18653/v1/2022.acl-long.58

    doi:10.18653/v1/2022.acl-long.58 2022

  11. [11]

    A. M. Mir and E. Latoskinas and G. Gousios , booktitle =. 2021 , pages =

    2021

  12. [12]

    arXiv preprint arXiv:2503.19786 , year=

    Gemma 3 technical report , author=. arXiv preprint arXiv:2503.19786 , year=

    Pith/arXiv arXiv

  13. [13]

    ROUGE : A Package for Automatic Evaluation of Summaries

    Lin, Chin-Yew. ROUGE : A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out. 2004

    2004

  14. [14]

    Curriculum learning , isbn =

    Bengio, Yoshua and Louradour, Jérôme and Collobert, Ronan and Weston, Jason , year =. Curriculum learning , isbn =. Proceedings of the 26th. doi:10.1145/1553374.1553380 , language =

    doi:10.1145/1553374.1553380

  15. [15]

    RWKV : Reinventing RNN s for the Transformer Era

    Peng, Bo and Alcaide, Eric and Anthony, Quentin and Albalak, Alon and Arcadinho, Samuel and Biderman, Stella and Cao, Huanqi and Cheng, Xin and Chung, Michael and Derczynski, Leon and Du, Xingjian and Grella, Matteo and Gv, Kranthi and He, Xuzheng and Hou, Haowen and Kazienko, Przemyslaw and Kocon, Jan and Kong, Jiaming and Koptyra, Bart omiej and Lau, Ha...

    2023

  16. [16]

    Advances in Neural Information Processing Systems , url =

    Attention is all you need , author=. Advances in Neural Information Processing Systems , url =

  17. [17]

    arXiv preprint arXiv:2009.03300 , year=

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    Pith/arXiv arXiv 2009

  18. [18]

    arXiv preprint arXiv:2310.06770 , year=

    Swe-bench: Can language models resolve real-world github issues? , author=. arXiv preprint arXiv:2310.06770 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    Pith/arXiv arXiv

  20. [20]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  21. [21]

    Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen

    Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen. LoRA: Low-Rank Adaptation of Large Language Models , journal =. 2021 , url =. 2106.09685 , timestamp =

    Pith/arXiv arXiv 2021

  22. [22]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  23. [23]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  24. [24]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  25. [25]

    Proceedings of the national academy of sciences , volume=

    Neural networks and physical systems with emergent collective computational abilities , author=. Proceedings of the national academy of sciences , volume=. 1982 , publisher=

    1982

  26. [26]

    Proceedings of the national academy of sciences , volume=

    Neurons with graded response have collective computational properties like those of two-state neurons , author=. Proceedings of the national academy of sciences , volume=. 1984 , publisher=

    1984

  27. [27]

    arXiv preprint arXiv:2310.02984 , year=

    Scaling laws for associative memories , author=. arXiv preprint arXiv:2310.02984 , year=

    Pith/arXiv arXiv

  28. [28]

    Associative Memory \ & \ Hopfield Networks in 2023 , year=

    A Different Route to Exponential Storage Capacity , author=. Associative Memory \ & \ Hopfield Networks in 2023 , year=

    2023

  29. [29]

    , booktitle =

    Krotov, Dmitry and Hopfield, John J. , booktitle =. Dense Associative Memory for Pattern Recognition , url =

  30. [30]

    On a Model of Associative Memory with Huge Storage Capacity , volume=

    Demircigil, Mete and Heusel, Judith and Löwe, Matthias and Upgang, Sven and Vermet, Franck , year=. On a Model of Associative Memory with Huge Storage Capacity , volume=. Journal of Statistical Physics , publisher=. doi:10.1007/s10955-017-1806-y , number=

    doi:10.1007/s10955-017-1806-y

  31. [31]

    arXiv preprint arXiv:2008.02217 , year=

    Hopfield networks is all you need , author=. arXiv preprint arXiv:2008.02217 , year=

    Pith/arXiv arXiv 2008

  32. [32]

    Advances in Neural Information Processing Systems , volume=

    Modern hopfield networks and attention for immune repertoire classification , author=. Advances in Neural Information Processing Systems , volume=

  33. [33]

    Cloob: Modern hopfield networks with infoloob outperform

    F. Cloob: Modern hopfield networks with infoloob outperform. Advances in Neural Information Processing Systems , volume=

  34. [34]

    Advances in Neural Information Processing Systems , volume=

    Energy transformer , author=. Advances in Neural Information Processing Systems , volume=

  35. [35]

    Ota, Toshihiro and Taki, Masato , journal=

  36. [36]

    International Conference on Learning Representations , year=

    Large Associative Memory Problem in Neurobiology and Machine Learning , author=. International Conference on Learning Representations , year=

  37. [37]

    A new frontier for

    Krotov, Dmitry , journal=. A new frontier for. 2023 , publisher=

    2023

  38. [38]

    International Conference on Learning Representations , year=

    Relating transformers to models and neural representations of the hippocampal formation , author=. International Conference on Learning Representations , year=

  39. [39]

    arXiv preprint arXiv:2304.14964 , year=

    The Exponential Capacity of Dense Associative Memories , author=. arXiv preprint arXiv:2304.14964 , year=

    Pith/arXiv arXiv

  40. [40]

    Onabola, Olawale and Ma, Zhuang and Xie, Yang and Akera, Benjamin and Ibraheem, Abdulrahman and Xue, Jia and Liu, Dianbo and Bengio, Yoshua , journal=

  41. [41]

    Frontiers in big Data , volume=

    Modern Hopfield Networks for graph embedding , author=. Frontiers in big Data , volume=. 2022 , publisher=

    2022

  42. [42]

    Advances in Neural Information Processing Systems , volume=

    Long sequence Hopfield memory , author=. Advances in Neural Information Processing Systems , volume=

  43. [43]

    International Conference on Machine Learning , pages=

    General sequential episodic memory model , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  44. [44]

    arXiv preprint arXiv:2305.11982 , year=

    Sequential Memory with Temporal Predictive Coding , author=. arXiv preprint arXiv:2305.11982 , year=

    Pith/arXiv arXiv

  45. [45]

    International Conference on Machine Learning , year=

    End-to-end Differentiable Clustering with Associative Memories , author=. International Conference on Machine Learning , year=

  46. [46]

    International Conference on Machine Learning , pages=

    Universal hopfield networks: A general framework for single-shot associative memory models , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  47. [47]

    The Eleventh International Conference on Learning Representations , year=

    Simplicial Hopfield networks , author=. The Eleventh International Conference on Learning Representations , year=

  48. [48]

    International Conference on Machine Learning , pages=

    Content addressable memory without catastrophic forgetting by heteroassociation with a fixed scaffold , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  49. [49]

    Nature , volume=

    Non-holographic associative memory , author=. Nature , volume=. 1969 , publisher=

    1969

  50. [50]

    IEEE Transactions on computers , volume=

    Learning patterns and pattern sequences by self-organizing nets of threshold elements , author=. IEEE Transactions on computers , volume=. 1972 , publisher=

    1972

  51. [51]

    arXiv preprint arXiv:2203.08913 , year=

    Memorizing transformers , author=. arXiv preprint arXiv:2203.08913 , year=

    Pith/arXiv arXiv

  52. [52]

    Advances in Neural Information Processing Systems , volume=

    Augmenting language models with long-term memory , author=. Advances in Neural Information Processing Systems , volume=

  53. [53]

    Proceedings of the AAAI Conference on Artificial Intelligence , url =

    Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory , author=. Proceedings of the AAAI Conference on Artificial Intelligence , url =

  54. [54]

    arXiv preprint arXiv:2402.13449 , year=

    CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory , author=. arXiv preprint arXiv:2402.13449 , year=

    Pith/arXiv arXiv

  55. [55]

    Nature Human Behaviour , pages=

    A generative model of memory construction and consolidation , author=. Nature Human Behaviour , pages=. 2024 , publisher=

    2024

  56. [56]

    arXiv preprint arXiv:2405.17822 , year=

    Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Action , author=. arXiv preprint arXiv:2405.17822 , year=

    Pith/arXiv arXiv

  57. [57]

    Neural Networks , volume=

    Dreaming neural networks: forgetting spurious memories and reinforcing pure ones , author=. Neural Networks , volume=. 2019 , publisher=

    2019

  58. [58]

    Physica A: Statistical Mechanics and its Applications , volume=

    Dense Hebbian neural networks: a replica symmetric picture of supervised learning , author=. Physica A: Statistical Mechanics and its Applications , volume=. 2023 , publisher=

    2023

  59. [59]

    arXiv preprint arXiv:2405.06394 , year=

    Memory Mosaics , author=. arXiv preprint arXiv:2405.06394 , year=

    Pith/arXiv arXiv

  60. [60]

    Advances in Neural Information Processing Systems , volume=

    Birth of a transformer: A memory viewpoint , author=. Advances in Neural Information Processing Systems , volume=

  61. [61]

    2024 , eprint=

    Nonparametric Modern Hopfield Models , author=. 2024 , eprint=

    2024

  62. [62]

    Forty-first International Conference on Machine Learning , year=

    Outlier-Efficient Hopfield Layers for Large Transformer-Based Models , author=. Forty-first International Conference on Machine Learning , year=

  63. [63]

    Workshop on Machine Learning and Compression, NeurIPS 2024 , year=

    Deep Clustering with Associative Memories , author=. Workshop on Machine Learning and Compression, NeurIPS 2024 , year=

    2024

  64. [64]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

    Dense Associative Memory Through the Lens of Random Features , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

  65. [65]

    2020 , eprint=

    Longformer: The Long-Document Transformer , author=. 2020 , eprint=

    2020

  66. [66]

    Transactions of the Association for Computational Linguistics , url =

    Lost in the middle: How language models use long contexts , author=. Transactions of the Association for Computational Linguistics , url =

  67. [67]

    Recurrent Memory Transformer , url =

    Bulatov, Aydar and Kuratov, Yury and Burtsev, Mikhail , booktitle =. Recurrent Memory Transformer , url =

  68. [68]

    Transformers are

    Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, Fran. Transformers are. International Conference on Machine Learning , url=

  69. [69]

    Jongho Park and Jaeseung Park and Zheyang Xiong and Nayoung Lee and Jaewoong Cho and Samet Oymak and Kangwook Lee and Dimitris Papailiopoulos , booktitle=. Can

  70. [70]

    arXiv preprint arXiv:2502.02737 , year =

    Loubna Ben Allal and Anton Lozhkov and Elie Bakouch and Gabriel Mart. arXiv preprint arXiv:2502.02737 , year =. doi:10.48550/ARXIV.2502.02737 , eprinttype =. 2502.02737 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.02737

  71. [71]

    Penedo, Guilherme and Kydl\'. The. Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =. 2024 , isbn =

    2024

  72. [72]

    arXiv preprint arXiv:2403.05530 , year=

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context , author=. arXiv preprint arXiv:2403.05530 , year=

    Pith/arXiv arXiv

  73. [73]

    Introducing the next generation of

    Anthropic , year = 2024, howpublished =. Introducing the next generation of

    2024

  74. [74]

    New models and developer products announced at

    OpenAI , year = 2023, howpublished =. New models and developer products announced at

    2023

  75. [75]

    Zaheer, Manzil and Guruganesh, Guru and Dubey, Kumar Avinava and Ainslie, Joshua and Alberti, Chris and Ontanon, Santiago and Pham, Philip and Ravula, Anirudh and Wang, Qifan and Yang, Li and Ahmed, Amr , booktitle =

  76. [76]

    2020 , eprint=

    Linformer: Self-Attention with Linear Complexity , author=. 2020 , eprint=

    2020

  77. [77]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

    Dai, Zihang and Yang, Zhilin and Yang, Yiming and Carbonell, Jaime G and Le, Quoc and Salakhutdinov, Ruslan , url =. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages=

  78. [78]

    Proceedings of the 41st International Conference on Machine Learning , pages=

    Repeat after me: transformers are better than state space models at copying , author=. Proceedings of the 41st International Conference on Machine Learning , pages=

  79. [79]

    International Conference on Machine Learning , url =

    The Illusion of State in State-Space Models , author=. International Conference on Machine Learning , url =. 2024 , organization=

    2024

  80. [80]

    First Conference on Language Modeling , year=

    Mamba: Linear-time sequence modeling with selective state spaces , author=. First Conference on Language Modeling , year=

Showing first 80 references.