Pith. sign in

REVIEW 5 major objections 6 minor 36 references

Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection

T0 review · 5 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For state-space models, retrieved context can be injected as a precomputed hidden state in O(1), exactly matching in-context reading.

desk verdict PRECOG's exactness guarantee is a tautology, and the actual model's conv buffer breaks it; still, the state-injection idea is worth taking seriously. read the letter →

arxiv 2608.02560 v1 pith:LK3UAVZA submitted 2026-08-03 cs.LG cs.AIcs.IR

Anusha Madan Gopal , Aras Pirbadian , Kristofor D. Carlson , M Anthony Lewis , Jonathan Tapson This is my paper · ORCID
classification cs.LGcs.AIcs.IR
keywords state-spacemodelsretrieval-augmentedgenerationhiddenstateinjectionO(1)prefillpersistentmemoryedgeinferencetime-translationinvariancestate-basedRAG
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PRECOG is the claim that for any state-space language model with a time-translation-invariant recurrence, the retrieved-context phase of RAG can be moved offline: encode each chunk once into its fixed-size hidden state, then inject the best-matching state as the initial condition at query time. The paper proves this is not an approximation by showing the recurrence satisfies S(h0, c ⊕ q) = S(S(h0, c), q), so the injected-state trajectory is bit-identical to re-reading the chunk. On a 1.2-billion-parameter gated SSM with a 192 KB state, this cuts prefill from about 27 seconds to under 6 milliseconds on edge hardware while matching answer quality on extractive QA. The same primitive is extended into SMC, a hierarchical persistent memory with O(1) session initialization. If correct, recurrent models can do retrieval and long-term memory without paying any context-token ingestion cost, a structural advantage Transformers cannot replicate.

What carries the argument

The load-bearing object is Theorem 1, the PRECOG–RAG equivalence: for a time-translation-invariant SSM update Φ(h, x), S(h0, c ⊕ q) = S(S(h0, c), q). The property that makes it true is that the per-token update depends only on the current hidden state and token, so the fixed-size recurrent state is a position-agnostic sufficient statistic of everything read. This identity converts retrieval from a token-ingestion operation into a state-copy operation: store a 192 KB hidden state per chunk, retrieve by embedding similarity, and initialize the model's recurrent buffers with it. The paper argues the same identity fails for Transformer KV-caches because positional encodings make cache contents p

What would settle it

Run the same chunk through the model twice: once by prepending it to a query (in-context), once by saving the post-chunk state and injecting it. Compare the log-probabilities of the first generated token. Any deviation larger than the FP16 rounding bound (~2^-10 per element, accumulated over query length) disproves the exactness claim as implemented. The same test across chunk lengths from 100 to 3,000 tokens would separate a genuine memory-horizon effect from an unsaved auxiliary state: Theorem 1 predicts exact equality at every length, so a systematic divergence beyond roughly 600 tokens wou

Watch

Extended reading notes

Core claim

The central discovery is an algebraic identity with a deployment consequence: because the SSM update Φ(h, x) depends only on the current hidden state and token, not on absolute position, rolling the recurrence over a context and then a query equals rolling it over the concatenated sequence. Therefore a hidden state precomputed offline from a retrieved chunk is a sufficient statistic for that chunk; injecting it as the initial state reproduces, exactly, what the model would compute if it had read the chunk at the start of the query. The paper instantiates this on a 1.2B gated SSM, reports that top-1 injection matches in-context RAG within FP16 rounding on a 1,000-question extractive QA sample

Load-bearing premise

The saved state must be a complete snapshot of every memory-carrying buffer in the model—not just the recurrent state but also the causal convolution front-end, normalization statistics, and any gating or sampling registers—because the exactness identity holds only if the injected state is the state the model would have reached by reading the chunk.

Editorial extensions

If this is right

  • For any recurrent backbone whose update depends only on the current state and token, PRECOG turns retrieved-context ingestion into a single state copy; the answer distribution is identical to in-context RAG by construction.
  • On the paper's 1.2B gated SSM with a 192 KB state, prefill drops from about 27 seconds to under 6 ms on edge hardware, roughly a 4,500x speedup at matched answer quality on a 1,000-question extractive QA sample.
  • Per-chunk storage is constant in context length: 192 KB versus a KV cache that grows at 32 KB per token, crossing at 6 tokens and reaching an 85x gap at 512-token chunks, with the gap widening further at longer contexts.
  • The same injection substrate gives persistent device memory: consolidated semantic states are written into the recurrent state at session start, so initialization stays O(1) no matter how much history has accumulated.
  • The backbone's effective memory length bounds the useful chunk size: PRECOG and in-context RAG stay statistically indistinguishable below roughly 600 tokens and diverge beyond that because PRECOG inherits the model's forgetting profile exactly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the completeness condition is satisfied in deployed code, this design point extends beyond edge devices: any recurrent model serving hot chunks could trade roughly 200x more storage per chunk for a four-orders-of-magnitude reduction in per-query ingestion latency.
  • A straightforward engineering check follows from the theorem: compare first-token logits under injection versus in-context reading; any divergence above the FP16 rounding bound identifies an auxiliary state, such as a convolution buffer or normalization statistic, that was not saved.
  • The same algebraic identity suggests a route to exact multi-document retrieval: instead of averaging states in hidden space, one could search for a composition rule that respects the recurrence, or fine-tune retrieval to select a single state that already contains fused context.
  • The memory-horizon ablation implies a measurable deployment rule: chunk at the model's empirical forgetting length, and reserve in-context reading for the long tail, unless the backbone's effective memory is extended.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes PRECOG, a retrieval mechanism for SSM-based language models that pre-encodes corpus chunks offline into the model's recurrent hidden state and, at query time, injects the retrieved state as the initial condition instead of re-ingesting context tokens. The central formal claim is Theorem 1: for a time-translation-invariant SSM update Φ(h,x), rolling the recurrence over context then query equals rolling over the concatenated sequence. The authors instantiate this on TENNs-LLM, a 1.2B gated SSM with a 192 KB nominal hidden state, and report that top-1 PRECOG matches in-context RAG on SQuAD v1.1 while cutting the reported context-ingestion latency from ~27 s to <6 ms. A second mechanism, SMC, organizes accumulated interaction states into cognitive-domain clusters and uses the same injection primitive for persistent memory.

Significance. If the state-capture issue is resolved, the core insight is simple and appealing: for an SSM whose full recurrent state is saved and injectable, context ingestion at prefill becomes independent of retrieved-context length. The algebraic identity is proved correctly in the abstract setting, and the paper contains useful, reproducible storage/bandwidth calculations (Appendix D) plus a clean implementation-consistency experiment. However, the headline claims currently overreach: Theorem 1 is not connected to the actual stateful components of TENNs-LLM, the latency numbers are internally inconsistent, the main RAG experiment uses the gold paragraph in both arms, and one ablation directly contradicts the theorem. These issues are fixable, but they are load-bearing for the paper's claims.

major comments (5)
  1. [§3/Table 1, §4.2, §4.3, Appendix D.1] Theorem 1 is proved only for the abstract recurrence Φ(h,x). The actual TENNs-LLM block includes a causal convolution front-end (Table 1, kernel 4; Appendix E) whose buffer of previous input tokens is stateful. The artifact counted in Appendix D.1 (24×4096×2 = 192 KB) contains only the per-layer SSM recurrent state; it does not include the conv buffers (roughly (kernel−1)×d_inner×2 bytes per layer, i.e., ~576 KB if operating on the inner dimension). With an empty conv buffer after injection, the first query tokens in PRECOG see a different conv-filtered input than in in-context RAG, so the equality S(S(h0,c),q)=S(h0,c⊕q) is not established for the full model. The state definition must be expanded to include all stateful components, or the 'complete summary' claim must be withdrawn.
  2. [Abstract, Table 2, §4.2, Appendix D.6, Figure 3] The abstract and Table 2 state that TTFT/prefill is <6 ms, but Appendix D.6 decomposes PRECOG TTFT as 585 ms (5 ms retrieval, ~1 ms load/inject, 0.5 ms tokenization, 526 ms query ingestion at 19 tok/s, ~53 ms first-token compute). The <6 ms figure is only the overhead beyond normal query processing. Since the user-visible time to first token includes query processing, the headline '~27 s to <6 ms' is an order-of-magnitude overstatement. Please correct all such statements and consistently distinguish 'context-ingestion eliminated' from total TTFT. Figure 3 already shows 585 ms; the main text should match.
  3. [§6, Appendix F] The main SQuAD evaluation does not evaluate retrieval. Both arms use the gold paragraph: in-context RAG prepends the gold paragraph, and PRECOG injects the hidden state of that same paragraph. The reported 0.2 EM/F1 gap therefore tests only the state-injection implementation under Theorem 1; it says nothing about retrieval quality, false negatives, or the FAISS/embedding pipeline. A RAG paper needs an open-domain or distractor setting where the index is actually searched and retrieval recall/hits are reported. The HotpotQA ablation moves in this direction but is in an appendix and is not the headline result.
  4. [Appendix G.3, Table 8] Table 8 reports PRECOG losing up to 15 F1 to in-context RAG on chunks longer than ~600 tokens, with the text saying in-context RAG 'retains positional access to all tokens.' But both configurations use the same TENNs-LLM SSM backbone; under Theorem 1, in-context RAG after reading the same long paragraph also has only the fixed-size recurrent state and should have the same memory horizon. If the observation is real, then either the two arms differ in some unstated way (e.g., a different backbone or a non-recurrent path), or Theorem 1 does not hold for the full model. This appendix directly undermines the paper's central equivalence and must be reconciled.
  5. [§5, Appendix I.2] SMC's end-to-end claims—consolidation into semantic states, O(1) session initialization, and joint episodic/corpus retrieval—are supported only by cluster-separation ratios on Harry Potter film transcripts and a t-SNE visualization. Appendix I.2 explicitly states that full validation on naturalistic data with the deployed pipeline is left to future work. As written, SMC is an architecture proposal without task-level evidence. Include at least one quantitative downstream evaluation of memory-augmented QA or dialogue to substantiate the persistent-memory contribution.
minor comments (6)
  1. [§4.4, Eq. (8)] The softmax weights in Eq. (8) are written as softmax of similarities but no temperature or normalization detail is given. Clarify the exact composition rule used in the top-k experiments.
  2. [Table 3] The generation protocol uses top-p sampling (p=0.9), so the 0.2 EM/F1 difference between top-1 PRECOG and in-context RAG may be sampling noise. Report multiple seeds or confidence intervals before calling it a quantization-bound match.
  3. [§4.2, Figure 1] Figure 1 says 'first generated token at ~6 ms after retrieval,' which conflicts with the 585 ms TTFT in Appendix D.6. Make the figure and caption consistent with the corrected latency accounting.
  4. [Abstract, §4.1] The phrase 'complete summary of everything the model has read' is too strong; Proposition 1 gives a more precise statement (sufficient statistic for the continuation under model dynamics). Use the qualified phrasing throughout.
  5. [Appendix D.5] The appendix acknowledges mixing binary and decimal units; it would be cleaner to use one convention consistently in tables.
  6. [References] Some references carry 2026 arXiv identifiers (e.g., [9], [36] and the paper itself). Please verify all identifiers and dates are correct at publication time.

Circularity Check

3 steps flagged · score 6.0 of 10

PRECOG's 'matches in-context RAG' claim is Theorem 1 restated by construction; Theorem 1 is proven only for an abstract recurrence that omits TENNs-LLM's causal-conv buffer, and SMC extends the identity to an EMA state by naming.

  1. self definitional [Section 4.3; Appendix A (Theorem 1 / Theorem 2)]
    "The following identity is the load-bearing claim of the paper. Theorem 1 (PRECOG–RAG equivalence). For any initial state h0, context c, and query q, S(h0, c⊕q)=S(S(h0,c),q) ... The empirical claim 'PRECOG matches in-context RAG' is therefore guaranteed by construction; deviations larger than the quantization bound indicate implementation issues, not method failure."

    PRECOG's offline state is defined as S(h0,c), the same rollout the theorem compares against. The equality S(h0,c⊕q)=S(S(h0,c),q) follows immediately from the recurrence's dependence on (h,x) only; no parameter, dataset, or independent hypothesis enters. Thus the 'prediction' that PRECOG matches in-context RAG is the method's construction restated, and the SQuAD table is a consistency check of the copy operation rather than evidence for an independently derived forecast.

  2. other [Section 3 (Eq. 1) and Table 1 vs. Section 4.3 and Appendix D.1]
    "The update map Φ(h, x) := h⊙α(x) + β(x) depends on (h, x) only, with no explicit dependence on t. ... Full architectural details—TENNs block structure (RMSNorm, causal convolution front-end, gated residual path, output projection) ...—are in Appendix E. ... Total state = 24·4,096·2B = 196,608B = 192KB."

    Theorem 1 is stated and proved for the abstract map Φ(h,x), but the deployed TENNs block includes a causal-convolution front-end with kernel 4, which is itself stateful across the last kernel−1 tokens. The stored/injected 192 KB object covers only the recurrent SSM state, not the conv buffer. The derivation therefore reduces the full model to a recurrence it does not fully describe, stipulating equality at the abstract level rather than deriving it for TENNs-LLM; full-model equivalence is an assumption hidden in the theorem's abstraction.

1 more flagged steps
  1. self definitional [Section 5.3, Eq. (7)]
    "sm,j ←(1−α)sm,j + α ¯h(c), α∈(0,1]. ... By Theorem 1, this is equivalent to the model having ingested a consolidated history of prior interactions in the dominant domain—without ingesting a single context token at session start."

    Theorem 1 concerns exact sequences: only a state equal to S(h0, tokens) may be injected to reproduce those tokens. SMC's semantic state is an exponential moving average over final states of separate chunks, not the rollout over the concatenated interaction history. The assertion that injecting it 'is equivalent to the model having ingested a consolidated history' holds only if 'consolidated history' is defined to be the EMA state; otherwise it is an unproven approximation, not a consequence of the theorem.

full rationale

The central quality-match equivalence is an algebraic identity for the abstract SSM update: PRECOG stores S(h0,c) and injects it, and Theorem 1 is S(S(h0,c),q)=S(h0,c⊕q), which is the time-translation invariance/associativity of the recurrence itself. Consequently Table 3 is an implementation sanity check rather than independent empirical support. The load-bearing reduction is sharpened by the causal-conv front-end: Table 1 lists kernel 4 and Appendix E mentions the conv front-end, while the saved state (Appendix D.1) is only 24×4096×2B; neither Φ in Theorem 1 nor the 192 KB state includes the conv buffer, so exactness for the full TENNs-LLM is unproven. SMC's session initialization uses an EMA of separate chunk states (Eq. 7) and then invokes Theorem 1 to call it 'a consolidated history,' which is a definitional renaming rather than a derived consequence. No fitted parameter is relabeled as a prediction, and the top-k composition is openly admitted to be heuristic; the only author self-citation ([36], a pending patent) is non-load-bearing. The paper has substantial independent engineering content in latency, storage, and deployment, so this is not a pure fit-to-data circularity. Score 6 reflects that the central 'matches in-context RAG' prediction reduces by construction and that the theorem-to-model step is stipulated by omitting auxiliary state.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The paper's exactness claim relies on the domain assumption of a complete, position-agnostic SSM state; the system-level retrieval adds an external sentence-encoder assumption that is never validated in a realistic retrieval evaluation. Design choices such as L_chunk, k, alpha, and the prompt are hand-set rather than fitted, but several are unconstrained in the text.

free parameters (5)
  • chunk length L_chunk = 512 tokens
    Selected for KV-cache comparison; affects storage footprint and memory horizon; not fitted but a design choice central to evaluation.
  • top-k retrieval count k = 3 default
    Chosen by hand; ablation shows optimal k is corpus-dependent (k=1 for SQuAD, k=2-3 for HotpotQA).
  • SMC EMA coefficient alpha = not reported
    Exponential moving average in Eq. 7; no value or schedule given; determines semantic-state update.
  • cluster routing threshold = 0.2
    Routing probability threshold in Appendix I.2; ad hoc.
  • prompt template = ###{question} ###Long Answer:
    Manual design choice; affects generated answers.
assumptions (4)
  • domain assumption SSM update map Phi is time-translation invariant, depending only on current hidden state and token, with no absolute position dependence.
    Eq. (1) and Section 4.3 state this; needed for Theorem 1. Holds for Mamba-style selective SSMs in principle, but must include all auxiliary state.
  • domain assumption Stored 24x4096 hidden-state tensor is a complete summary of all model memory at the chunk boundary.
    Required for injection exactness; causal-conv buffers and normalization state are not accounted for in the stated 192 KB footprint.
  • domain assumption Sentence-encoder cosine similarity (all-MiniLM-L12-v2) retrieves chunks whose injected states improve answers.
    Retrieval index operation is asserted; the paper's experiments use gold paragraphs, so retrieval quality is not measured.
  • standard math Induction on query length |q| is valid.
    Standard proof technique used in Appendix A; not controversial.
invented entities (2)
  • SMC cognitive-domain taxonomy (EMOTIONAL, TEMPORAL, SOCIAL, SPATIAL, FACTUAL)
    purpose: Organizes accumulated episodic states into hierarchical clusters for constant-time session initialization.
    Validated only on Harry Potter dialogue with GPT-4o-generated prototypes; no end-to-end memory task and no external falsifiable prediction.
  • Emergent sub-cluster detection in the 'Others' bucket
    purpose: Allows runtime extension of the fixed taxonomy when unmatched chunks form dense groups.
    Qualitative t-SNE observation on one fictional corpus; no reproducible criterion or predictive test released.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection." pith.science (2026). https://pith.science/paper/LK3UAVZA

@misc{pith2026260802560,
  author       = {Pith},
  title        = {Pith review of: Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LK3UAVZA}},
  note         = {Machine review of arXiv:2608.02560}
}
abstract

Retrieval-augmented generation (RAG) imposes a prefill cost proportional to retrieved context length, and -- with Transformer backbones -- a KV-cache that grows with each generated token. State-Space Models (SSMs) avoid the second cost by construction; we eliminate the first, collapsing prefill from $O(L_{context})$ to $O(1)$ per query. We introduce PRECOG (Pre-Computed Context Injection), a retrieval mechanism that exploits a property unique to SSMs: the fixed-size, position-agnostic recurrent hidden state is a complete summary of everything the model has read. PRECOG pre-encodes document corpora offline as SSM hidden states and injects the best-matching state directly at query time, bypassing in-context re-ingestion entirely. The same state-injection mechanism enables SMC (Structured Memory Consolidation): a hierarchical persistent memory with cognitive-domain clustering, an adjustable fidelity-vs-storage dial, and $O(1)$ session initialization, which consolidates short-term episodic states into long-term semantic memory and fuses both with retrieved corpus states at query time. We demonstrate the system on TENNs-LLM, a 1.2B-parameter gated-SSM language model with a 192 KB hidden state. PRECOG matches in-context RAG answer quality, reducing prefill latency from $\sim$27 s to $<$6 ms on edge hardware -- a $\sim$4500$\times$ speedup that crosses the threshold from unusable to interactive. The mechanism is architecturally impossible for Transformer KV-caches, which are position-entangled and grow linearly with context length.

Figures

Figures reproduced from arXiv: 2608.02560 by the authors.

Figure 1
Figure 1. The PRECOG pipeline. Offline indexing (top): each chunk is encoded once by the SSM into a hidden state h(c), paired with a sentence-encoder key. Query time (bottom): the query is encoded; the top-k states (k=3 default) are retrieved by similarity, composed via softmax-weighted averaging into hinit, and injected as the initial recurrent state. The model processes only query tokens, producing the first generated token… view at source ↗
Figure 2
Figure 2. Per-chunk storage vs. context length (log–log). The Llama-3.2-1B KV-cache grows at [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Time from query arrival to first generated token, log time axis. UFS 4.0 storage and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Bandwidth-bound load time for the chunk artifact at [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Bandwidth-bound generation throughput as a function of active context length. PRE [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: The storage–latency frontier of state-level retrieval. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Structured Memory Consolidation pipeline. Each conversation chunk is encoded by [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: t-SNE 2-D projection of SPATIAL-domain dialogue chunks from Harry Potter film 7 (held￾out episodic test set), routed against sub-cluster prototypes built from films 1–6. Colored regions correspond to the ten predefined sub-clusters; gray points labeled “Others” (657 ch…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 7 linked inside Pith

  1. [1]

    Efficiently modeling long sequences with structured state spaces,

    A. Gu, K. Goel, and C. Ré, “Efficiently modeling long sequences with structured state spaces,”ICLR, 2022

    2022

  2. [2]

    Simplified state space layers for sequence modeling,

    J. T. H. Smith, A. Warrington, and S. W. Linderman, “Simplified state space layers for sequence modeling,” ICLR, 2023

    2023

  3. [3]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,”COLM, 2024. arXiv:2312.00752

    arXiv 2024

  4. [4]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality,

    T. Dao and A. Gu, “Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality,”ICML, 2024

    2024

  5. [5]

    RWKV: Reinventing RNNs for the transformer era,

    B. Peng et al., “RWKV: Reinventing RNNs for the transformer era,”Findings of EMNLP, 2023

    2023

  6. [6]

    Retentive network: A successor to transformer for large language models,

    Y . Sun, L. Dong, S. Huang, S. Ma, Y . Xia, J. Xue, J. Wang, and F. Wei, “Retentive network: A successor to transformer for large language models,”arXiv:2307.08621, 2023

    arXiv 2023

  7. [7]

    State soup: In-context skill learning, retrieval and mixing,

    M. Pióro, M. Wołczyk, R. Pascanu, J. von Oswald, and J. Sacramento, “State soup: In-context skill learning, retrieval and mixing,”arXiv:2406.08423, 2024

    arXiv 2024

  8. [8]

    PICASO: Permutation-invariant context composition with state space models,

    T. Y . Liu, A. Achille, M. Trager, A. Golatkar, L. Zancato, and S. Soatto, “PICASO: Permutation-invariant context composition with state space models,”ICLR, 2025

    2025

Show all 36 references
  1. [9]

    Memory caching: RNNs with growing memory,

    A. Behrouz, Z. Li, Y . Deng, P. Zhong, M. Razaviyayn, and V . Mirrokni, “Memory caching: RNNs with growing memory,”arXiv:2602.24281, 2026

    2026

  2. [10]

    Mistral 7B,

    A. Q. Jiang et al., “Mistral 7B,”arXiv:2310.06825, 2023

    2023 arXiv

  3. [11]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,”ICLR, 2022

    2022

  4. [12]

    Retrieval-augmented generation for knowledge-intensive NLP tasks,

    P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,”NeurIPS, 2020

    2020

  5. [13]

    Leveraging passage retrieval with generative models for open domain question answering,

    G. Izacard and E. Grave, “Leveraging passage retrieval with generative models for open domain question answering,”EACL, 2021

    2021

  6. [14]

    Improving language models by retrieving from trillions of tokens,

    S. Borgeaud et al., “Improving language models by retrieving from trillions of tokens,”ICML, 2022

    2022

  7. [15]

    Atlas: Few-shot learning with retrieval augmented language models,

    G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, and E. Grave, “Atlas: Few-shot learning with retrieval augmented language models,”JMLR, vol. 24, no. 251, pp. 1–43, 2023

    2023

  8. [16]

    REPLUG: Retrieval-augmented black-box language models,

    W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, and W.-t. Yih, “REPLUG: Retrieval-augmented black-box language models,”NAACL, 2024

    2024

  9. [17]

    xRAG: Extreme context compression for retrieval-augmented generation with one token,

    X. Cheng et al., “xRAG: Extreme context compression for retrieval-augmented generation with one token,” NeurIPS, 2024

    2024

  10. [18]

    LLMLingua: Compressing prompts for accelerated inference of large language models,

    H. Jiang, Q. Wu, C.-Y . Lin, Y . Yang, and L. Qiu, “LLMLingua: Compressing prompts for accelerated inference of large language models,”EMNLP, 2023

    2023

  11. [19]

    Adapting language models to compress contexts,

    A. Chevalier, A. Wettig, A. Ajith, and D. Chen, “Adapting language models to compress contexts,”EMNLP, 2023

    2023

  12. [20]

    Learning to compress prompts with gist tokens,

    J. Mu, X. L. Li, and N. Goodman, “Learning to compress prompts with gist tokens,”NeurIPS, 2023

    2023

  13. [21]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” EMNLP, 2021

    2021

  14. [22]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,”ACL-IJCNLP, 2021

    2021

  15. [23]

    Activation addition: Steering language models without optimization,

    A. M. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and M. MacDiarmid, “Activation addition: Steering language models without optimization,”arXiv:2308.10248, 2023

    2023 arXiv

  16. [24]

    In-context vectors: Making in-context learning more effective and controllable through latent space steering,

    S. Liu, H. Ye, L. Xing, and J. Zou, “In-context vectors: Making in-context learning more effective and controllable through latent space steering,”ICML, 2024

    2024

  17. [25]

    Human-inspired episodic memory for infinite context LLMs,

    Z. Fountas, M. A. Benfeghoul, A. Oomerjee, F. Christopoulou, G. Lampouras, H. Bou-Ammar, and J. Wang, “Human-inspired episodic memory for infinite context LLMs,”ICLR, 2025

    2025

  18. [26]

    Episodic and semantic memory,

    E. Tulving, “Episodic and semantic memory,” inOrganization of Memory, E. Tulving and W. Donaldson, Eds. New York: Academic Press, 1972, pp. 381–403

    1972

  19. [27]

    Efficient memory management for large language model serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,”SOSP, 2023

    2023

  20. [28]

    FlashAttention: Fast and memory-efficient exact attention with IO-awareness,

    T. Dao, D. Y . Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness,”NeurIPS, 2022

    2022

  21. [29]

    Textbooks are all you need II: phi-1.5 technical report,

    Y . Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y . T. Lee, “Textbooks are all you need II: phi-1.5 technical report,”arXiv:2309.05463, 2023. 10

    2023 arXiv

  22. [30]

    MobileLLM: Optimizing sub-billion parameter language models for on-device use cases,

    Z. Liu et al., “MobileLLM: Optimizing sub-billion parameter language models for on-device use cases,” ICML, 2024

    2024

  23. [31]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama,

    D. Soboleva, F. Al-Khateeb, R. Myers, J. R. Steeves, J. Hestness, and N. Dey, “SlimPajama: A 627B token cleaned and deduplicated version of RedPajama,” 2023. https://huggingface.co/datasets/ cerebras/SlimPajama-627B

    2023

  24. [32]

    The Pile: An 800GB dataset of diverse text for language modeling,

    L. Gao et al., “The Pile: An 800GB dataset of diverse text for language modeling,”arXiv:2101.00027, 2020

    2020 arXiv

  25. [33]

    SQuAD: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,”EMNLP, 2016

    2016

  26. [34]

    HotpotQA: A dataset for diverse, explainable multi-hop question answering,

    Z. Yang, P. Qi, S. Zhang, Y . Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning, “HotpotQA: A dataset for diverse, explainable multi-hop question answering,” EMNLP, 2018

    2018

  27. [35]

    Natural Questions: A benchmark for question answering research,

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov, “Natural Questions: A benchmark for question answering rese...

    2019

  28. [36]

    System and Method for Efficient Execution of Large Generative Artificial Intelligence Models on Edge Devices Using State-Space Models,

    M. A. Lewis, Y . R. Pei, J. Tapson, and A. Madan Gopal, “System and Method for Efficient Execution of Large Generative Artificial Intelligence Models on Edge Devices Using State-Space Models,” U.S. Patent Application Publication No. US 2026/0072920 A1, filed September 10, 2025...

    2026

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.