Pith. sign in

REVIEW 4 major objections 7 minor 35 references

Frozen LLM gains knowledge from SSD-backed phrase memory, no training

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 12:16 UTC pith:HTYDIFNS

load-bearing objection A train-free phrase-memory system for LLMs with SSD-backed storage and early-exit prefetching — the system engineering is the real contribution; the accuracy gains are modest and the injection mechanism is under-specified. the 4 major comments →

arxiv 2607.07388 v1 pith:HTYDIFNS submitted 2026-07-08 cs.CL cs.LG

TF-Engram: A Train-Free Engram with SSD-Backed Memory for Large Language Models

Yutang Ma , Kecheng Huang , Xikun Jiang , Zili Shao This is my paper
classification cs.CL cs.LG
keywords memorylargetf-engramsystemtrain-freeengramexternalknowledge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

TF-Engram argues that a large language model can acquire new factual knowledge without any parameter updates by injecting phrase-level semantic vectors drawn from an external memory table stored across GPU, DRAM, and NVMe SSD. The paper's central claim is that this train-free, storage-backed memory pathway improves downstream task accuracy on a 0.6B-parameter model (Qwen3) from 57.6 to 59.4 on average, outperforming a parameter-matched LoRA baseline at 58.7, while keeping the backbone frozen. The system avoids the hash-collision problem of prior compact memory designs by preserving one distinct vector per mined phrase and offloading the capacity burden to a storage hierarchy. To hide the latency of SSD and DRAM access during token-by-token generation, TF-Engram attaches an auxiliary prediction head to an intermediate Transformer layer, which forecasts likely upcoming phrases and triggers asynchronous prefetching before the full forward pass completes.

Core claim

The paper demonstrates that static phrase-level memory, constructed offline from corpora using a frozen embedding encoder and stored on commodity SSDs, can be integrated into LLM inference through gated residual injection into hidden states, yielding measurable accuracy gains on knowledge-intensive benchmarks (MMLU, ARC-Challenge, TruthfulQA, BoolQ) without any gradient-based training of the backbone. The early-exit guided prefetching mechanism recovers much of the throughput lost to external memory access by overlapping data movement with the computation of the remaining Transformer layers.

What carries the argument

Three components carry the argument: (1) offline phrase mining and frozen-encoder embedding construction that produces one vector per phrase without training; (2) a GPU-DRAM-SSD three-tier storage hierarchy that holds the full phrase table while caching hot entries on-device; (3) an early-exit prefetch head attached at layer L-r that predicts top-K future tokens and issues asynchronous memory fetches while layers L-r+1 through L continue computing. Memory injection uses gated residual addition: h'_t = h_t + g_t * m_t, where m_t is the retrieved phrase embedding and g_t is a gate.

Load-bearing premise

The memory injection mechanism assumes that phrase embeddings produced by a frozen external encoder (Qwen3-Embedding) are semantically compatible with the backbone LLM's hidden state space when added via gated residual injection, without training a projection between the two spaces. If the injected vectors are not in a compatible subspace, the gate may suppress them or they may add noise rather than useful signal, which is consistent with the modest gains and degradation on H

What would settle it

If the injected phrase embeddings from the external encoder occupy a subspace poorly aligned with the backbone's hidden states, the gate should systematically suppress injected memory signals, and accuracy gains should vanish or reverse when the injection scale is increased beyond a narrow band. The degradation on HellaSwag and WinoGrande already hints at this partial incompatibility.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper proposes TF-Engram, a train-free external memory system for LLMs that constructs phrase-specific semantic memory offline from corpora, stores it across a GPU–DRAM–SSD hierarchy, and uses early-exit guided predictive prefetching to hide external memory latency during autoregressive decoding. On Qwen3-0.6B, TF-Engram improves the average downstream score from 57.6 to 59.4 across ten benchmarks, outperforming a parameter-matched LoRA baseline (58.7). The paper also provides overhead analysis (construction cost, runtime memory, throughput, latency) and sensitivity studies on memory scale and early-exit layer position.

Significance. The paper makes a genuine systems contribution in externalizing Engram-style memory to an SSD-backed hierarchy with early-exit prefetching. The latency-hiding design—using an intermediate-layer prediction to prefetch from SSD while remaining Transformer layers compute—is a practical and well-motivated idea. The overhead analysis (Figures 3–5) is reasonably thorough, showing that a 100M-entry table can be built in ~2 hours and that prefetching recovers a meaningful fraction of throughput loss. The end-to-end evaluation on ten benchmarks with a parameter-matched LoRA baseline is a fair comparison design. The train-free construction pipeline (offline phrase mining, frozen encoder, no backbone training) is a clear and falsifiable design point.

major comments (4)
  1. §4.2, Eq. (5): The injection gate g_t is never specified in the implementation. The paper states it 'may be a scalar, a vector, or a learned function' (§2.2), but since the system is train-free, g_t cannot be learned. §5.4 lists 'memory injection scale' as a tunable parameter, which implies g_t is a fixed scalar α, making the mechanism h'_t = h_t + α·e_i. This should be stated explicitly. More importantly, e_i is produced by Qwen3-Embedding (Eq. 9), a contrastive embedding model whose representation space is shaped by a different objective than Qwen3-0.6B's hidden states (shaped by causal LM training). The paper provides no projection layer, no alignment analysis, and no evidence that these spaces are compatible. This is load-bearing for the central quality-improvement claim: the modest gains (1.8 points) and the degradations on HellaSwag (−0.4) and WinoGrande (−0.2) are consistent with,
  2. §4.4, Eq. (13): The 'train-free' claim is ambiguous with respect to the early-exit prefetch head W_pref. The paper presents two options—a frozen projection reused from the backbone ('strict train-free') and a lightly calibrated W_pref trained on unlabeled text—but does not clearly state which is used in the main experiments (Table 3). If W_pref is calibrated, the 'train-free' framing in the title and abstract is partially inaccurate for the reported results. The paper should state unambiguously which variant produced the Table 3 numbers and, if calibration was used, quantify its cost and revise the framing accordingly.
  3. §5.2, Table 3: The LoRA baseline is described only as 'parameter-matched' with a 'comparable trainable-parameter budget.' The paper does not specify the training data, learning rate, number of epochs, or task formulation used for LoRA. Since the central comparison (TF-Engram vs. LoRA, +0.7 average) is a key claim, the LoRA training setup must be specified in sufficient detail for reproduction. Without this, it is unclear whether LoRA was trained on the same corpora used for TF-Engram's memory construction or on task-specific data.
  4. §5: Only one backbone model (Qwen3-0.6B) is evaluated. While the system-level contributions (SSD hierarchy, prefetching) may generalize, the quality-improvement claim is tested on a single small model. The paper should either test at least one additional backbone (e.g., a 1.5B–3B model) or explicitly scope the quality claim to Qwen3-0.6B and discuss why results may or may not transfer.
minor comments (7)
  1. §5.4: The sensitivity analysis text for 'Impact of memory scale' and 'Impact of early-exit layer position' appears truncated in the manuscript (e.g., 'from 1.12' and 'a 3.20' without completing the sentences). Please fix.
  2. §4.2, Eq. (11): The aggregation weights α_i are described as determined by 'phrase length, frequency, or confidence' but the specific formula or selection rule is not given. A brief specification would help reproducibility.
  3. §4.4, Eq. (17): The priority score s_i = p_i · r_i · c_i uses three factors, but r_i ('usefulness of the phrase entry') and c_i ('expected benefit of prefetching') are not formally defined. Please specify how these are computed.
  4. Figure 1 is referenced but the caption is minimal. Consider adding labels or a brief walkthrough of the data flow (offline construction → hierarchy → online prefetch) directly in the caption.
  5. §5.3, Figure 5: The latency overhead percentages appear truncated ('a 3.83' and 'from 9.57'). Please complete these values.
  6. The paper would benefit from a brief discussion of how phrase matching works during autoregressive decoding—specifically, at what point in the token generation loop does the system check for phrase matches, and how does it handle partial matches across generation steps?
  7. Table 2 lists '516GB' SSD storage, which is an unusual capacity. Please verify this is correct (common NVMe sizes are 512GB or 1TB).

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises four major points: (1) the injection gate g_t and the embedding-space compatibility between Qwen3-Embedding and Qwen3-0.6B hidden states are underspecified; (2) the 'train-free' framing is ambiguous with respect to the early-exit prefetch head W_pref; (3) the LoRA baseline lacks reproducible training details; and (4) only one backbone model is evaluated. We agree that all four points identify genuine gaps in the current manuscript. We will revise the paper to explicitly state the gate formulation, add alignment analysis, clarify which prefetch-head variant was used in main experiments, fully specify the LoRA training setup, and add a second backbone. Details follow.

read point-by-point responses

  1. Referee: §4.2, Eq. (5): The injection gate g_t is never specified in the implementation. The paper states it 'may be a scalar, a vector, or a learned function' (§2.2), but since the system is train-free, g_t cannot be learned. §5.4 lists 'memory injection scale' as a tunable parameter, which implies g_t is a fixed scalar α, making the mechanism h'_t = h_t + α·e_i. This should be stated explicitly. More importantly, e_i is produced by Qwen3-Embedding (Eq. 9), a contrastive embedding model whose representation space is shaped by a different objective than Qwen3-0.6B's hidden states (shaped by causal LM training). The paper provides no projection layer, no alignment analysis, and no evidence that these spaces are compatible. This is load-bearing for the central quality-improvement claim: the modest gains (1.8 points) and the degradations on HellaSwag (−0.4) and WinoGrande (−0.2) are consistent with,

    Authors: The referee is correct on both points. In our implementation, g_t is indeed a fixed scalar α, so the injection mechanism is h'_t = h_t + α·e_i. We will state this explicitly in §4.2 and remove the ambiguous language about learned gates. Regarding the embedding-space compatibility concern: this is a fair and important observation. In the revision we will add an analysis of the geometric relationship between Qwen3-Embedding vectors and Qwen3-0.6B hidden states, including cosine-similarity distributions and a comparison of nearest-neighbor overlap. We acknowledge that the absence of a projection layer is a limitation: the embedding and hidden-state spaces are not explicitly aligned, and the modest gains and per-task degradations (HellaSwag, WinoGrande) are consistent with partial incompatibility. We will discuss this limitation honestly and note that a learned projection layer (which would require a small calibration step) is a natural extension, though it would partially weaken the strict train-free claim. We will also report results with a simple untrained linear projection (e.g., mean-and-variance matching) as an ablation to quantify the impact. revision: yes

  2. Referee: §4.4, Eq. (13): The 'train-free' claim is ambiguous with respect to the early-exit prefetch head W_pref. The paper presents two options—a frozen projection reused from the backbone ('strict train-free') and a lightly calibrated W_pref trained on unlabeled text—but does not clearly state which is used in the main experiments (Table 3). If W_pref is calibrated, the 'train-free' framing in the title and abstract is partially inaccurate for the reported results. The paper should state unambiguously which variant produced the Table 3 numbers and, if calibration was used, quantify its cost and revise the framing accordingly.

    Authors: The referee is correct that the manuscript does not unambiguously state which variant was used for the Table 3 results. We will fix this. In the main experiments (Table 3), we used the strict train-free variant: W_pref is a frozen projection derived from the backbone model's own intermediate-layer-to-vocabulary projection, without any calibration or gradient updates. No training of any kind was performed on the backbone, the memory table, or the prefetch head for the reported results. We will state this explicitly in §4.4 and §5.1. We will also report the calibrated variant as a supplementary ablation, including its training cost (amount of unlabeled text, training time, and parameter count) so that readers can assess the trade-off. The title and abstract framing of 'train-free' is accurate for the main results; we will make this unambiguous in the revision. revision: yes

  3. Referee: §5.2, Table 3: The LoRA baseline is described only as 'parameter-matched' with a 'comparable trainable-parameter budget.' The paper does not specify the training data, learning rate, number of epochs, or task formulation used for LoRA. Since the central comparison (TF-Engram vs. LoRA, +0.7 average) is a key claim, the LoRA training setup must be specified in sufficient detail for reproduction. Without this, it is unclear whether LoRA was trained on the same corpora used for TF-Engram's memory construction or on task-specific data.

    Authors: The referee is correct. The LoRA baseline description is insufficient for reproduction. We will add a full specification in §5.1, including: rank and alpha, target modules, training data (the same corpus mixture used for TF-Engram memory construction: FineWeb-Edu, Wikipedia, C4, arXiv, PubMed), learning rate, batch size, number of epochs, optimizer, warmup schedule, and sequence length. The LoRA was trained on the same corpora used for TF-Engram's memory construction (not task-specific data), making the comparison conservative: LoRA had access to the same knowledge source as TF-Engram's memory table. We will also report the exact trainable parameter count for both LoRA and the TF-Engram memory pathway to substantiate the 'parameter-matched' claim. If, upon re-running with fully specified hyperparameters, the LoRA numbers change materially, we will update Table 3 accordingly. revision: yes

  4. Referee: §5: Only one backbone model (Qwen3-0.6B) is evaluated. While the system-level contributions (SSD hierarchy, prefetching) may generalize, the quality-improvement claim is tested on a single small model. The paper should either test at least one additional backbone (e.g., a 1.5B–3B model) or explicitly scope the quality claim to Qwen3-0.6B and discuss why results may or may not transfer.

    Authors: The referee is correct that evaluating on a single backbone is a limitation. We will address this by adding experiments with at least one additional backbone in the 1.5B–3B range (e.g., Qwen3-1.7B). This will test whether the quality improvement transfers to a larger model and whether the embedding-space compatibility concern (raised in the first comment) becomes more or less severe with scale. We will report the full Table 3 benchmark suite for the additional backbone. If the gains do not transfer, we will explicitly scope the quality claim to Qwen3-0.6B and discuss possible reasons (e.g., larger models may already encode phrase-level knowledge that a 0.6B model lacks, reducing the marginal benefit of external memory). The system-level contributions (SSD hierarchy, prefetching) are architecture-agnostic and should transfer regardless, but the quality claim needs empirical support at additional scales. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; the derivation chain is self-contained against external benchmarks.

full rationale

The paper's central claim — that phrase-specific semantic memory improves downstream accuracy on a frozen backbone — does not reduce to its inputs by construction. The memory table is built offline from external corpora (FineWeb-Edu, Wikipedia, C4, arXiv, PubMed) using a frozen external encoder (Qwen3-Embedding, Eq. 9), and the backbone (Qwen3-0.6B) is never trained on the target benchmarks. The injection mechanism (Eq. 5: h'_t = h_t + g_t · m_t) adds externally-derived vectors to hidden states; while the gate g_t is underspecified (apparently a fixed scalar tuned via §5.4 sensitivity analysis), this is a correctness/alignment concern, not a circularity — the injected vectors come from a different model and corpus, not from the backbone's own representations. The early-exit prefetch head (Eq. 13) explicitly does not affect final token probabilities ('the early-exit head affects only prefetch scheduling and does not change the final token probabilities'). Hyperparameters (memory injection scale, early-exit layer position, top-K budget) are tuned on the evaluation benchmarks in §5.4, which is a methodological concern about train/test separation but not formal circularity — the paper does not claim these tuned values as first-principles predictions. No self-citation chain load-bears the central claim: the key prior work citations ([5], [6]) are by different author groups. The modest gains (1.8 points average, with degradation on HellaSwag/WinoGrande) are consistent with an independent (if weak) signal rather than a construction-forced result. Score 1 reflects the minor concern that hyperparameter selection on benchmark data slightly weakens the 'train-free' framing, but the core derivation has independent content.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 1 invented entities

The system introduces 4 free parameters tuned on evaluation benchmarks, 1 domain assumption about embedding-hidden state compatibility that is load-bearing but unverified, and 1 auxiliary component (early-exit head) whose necessity is not ablated against simpler alternatives. The axioms are mostly standard for the memory-augmented LLM literature.

free parameters (4)
  • Memory injection scale (gate magnitude) = Not explicitly stated; tuned in sensitivity analysis (§5.4)
    Controls how strongly retrieved memory modifies hidden states (Eq. 5). Selected based on downstream performance.
  • Early-exit layer position (r) = Layer 8 of Qwen3-0.6B (§5.4)
    Determines prediction quality vs. latency-hiding window trade-off. Selected post-hoc via sensitivity sweep.
  • Top-K prefetch budget = Top-64 (§5.4)
    Controls how many candidate phrases are prefetched. Selected based on throughput-accuracy trade-off.
  • Phrase frequency threshold = Not specified
    Determines which n-grams from corpora become memory entries. Stated as a tunable parameter but value not given.
axioms (3)
  • domain assumption Phrase embeddings from a frozen external encoder (Qwen3-Embedding) are compatible with the backbone LLM's hidden state space for residual injection.
    §4.2, Eq. 5. The paper adds retrieved embeddings directly to hidden states without training a projection. No evidence of alignment is provided.
  • standard math Intermediate-layer hidden states contain sufficient predictive information to guide useful prefetching.
    §2.1, §4.4. Supported by prior work on early-exit and speculative decoding [4, 22].
  • domain assumption Static phrase-level memory provides useful signal for knowledge-intensive benchmarks.
    §3.1. The paper's evaluation partially confirms this (gains on MMLU, ARC) but also shows exceptions (HellaSwag, WinoGrande degradation).
invented entities (1)
  • Early-exit prefetch head (W_pref) no independent evidence
    purpose: Auxiliary projection at layer L-r to predict future token distribution for prefetch scheduling
    §4.4, Eq. 13. The paper describes it as optionally 'lightly calibrated' but does not provide a falsifiable prediction outside the system itself. It is a system component, not a new physical entity, but its necessity is not independently validated against alternatives (e.g., n-gram-based prefetch).

pith-pipeline@v1.1.0-glm · 23155 in / 2900 out tokens · 519210 ms · 2026-07-09T12:16:11.085940+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) store factual knowledge and domain-specific patterns implicitly in dense Transformer parameters, making knowledge expansion costly through pretraining, fine-tuning, retrieval augmentation, or longer contexts. Engram-style memory offers a compact hidden-state injection pathway, but existing GPU-resident designs often rely on hash-based compression, causing unrelated phrases to collide in shared slots and weakening phrase-level semantic fidelity. We present TF-Engram, a train-free Engram system that constructs phrase-specific semantic memory offline from external corpora, stores large memory tables across a GPU--DRAM--SSD hierarchy, and uses Early-Exit Guided Predictive Prefetching to hide external-memory latency during autoregressive decoding. On Qwen3-0.6B, TF-Engram improves the average downstream score from 57.6 to 59.4, outperforming both the frozen backbone and a parameter-matched LoRA baseline. System evaluation shows that large TF-Engram tables can be built with moderate offline cost, SSD-backed storage substantially reduces GPU memory demand, and predictive prefetching recovers much of the throughput loss caused by external memory access. These results demonstrate that static phrase memory can be integrated into LLM inference as a scalable, train-free, and low-overhead system component.

Figures

Figures reproduced from arXiv: 2607.07388 by Kecheng Huang, Xikun Jiang, Yutang Ma, Zili Shao.

Figure 1
Figure 1. Figure 1: Technique overview of TF-Engram. When multiple phrases match the current context, TF-Engram uses a deterministic selection policy. A simple policy is longest￾match selection, where longer phrases are preferred because they usually correspond to more complete semantic units. If several phrases have the same length, TF-Engram can use frequency, se￾mantic confidence, or corpus priority as tie-breakers. If mul… view at source ↗
Figure 2
Figure 2. Figure 2: Early-exit guided predictive prefetching. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Runtime resource overhead of TF-Engram [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Decode latency overhead of TF-Engram. footprint, increasing GPU memory usage to 24.03 GB. In contrast, the SSD-backed 100M-entry table reduces GPU memory usage to 3.06 GB, while shifting the storage burden to CPU memory and SSD-backed access. This demonstrates the main benefit of the SSD￾backed design: TF-Engram can support a much larger memory table without keeping the full table in GPU memory. The SSD-ba… view at source ↗
Figure 6
Figure 6. Figure 6: Sensitivity analysis of TF-Engram. 5.4 Sensitivity Analysis We further conduct a sensitivity analysis to understand how TF￾Engram behaves under different memory scales and early-exit layer positions. The results are shown in [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 35 canonical work pages · 10 internal anchors

  1. [1]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. 2022. DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. InProceedings of the International Conference for High Performance Computing, Networking,...

    work page 2022

  2. [2]

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Ruther- ford, Katie Millican, George B. M. van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irv...

    work page 2022

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page 2020

  4. [4]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Lau- rent Sifre, and John Jumper. 2023. Accelerating Large Language Model Decoding with Speculative Sampling.arXiv preprint arXiv:2302.01318(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    Runxi Cheng, Yuchen Guan, Yongxian Wei, Qianpu Sun, Qixiu Li, Sinan Du, Feng Xiong, Chun Yuan, Yan Lu, and Yeyun Gong. 2026. Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory.arXiv preprint arXiv:2605.20948(2026)

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  6. [6]

    Xin Cheng, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, Kezhao Huang, Xingkai Yu, Zhewen Hao, Yukun Li, Han Zhang, Huishuai Zhang, Dongyan Zhao, and Wenfeng Liang. 2026. Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models.arXiv preprint arXiv:2601.07372(2026)

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  7. [7]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, et al. 2022. PaLM: Scaling Language Modeling with Pathways. arXiv preprint arXiv:2204.02311(2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  8. [8]

    Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. InProceedings of the International Conference on Learning Representations

    work page 2024

  9. [9]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R’e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R’e. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. InAdvances in Neural Information Processing Systems

    work page 2022

  10. [10]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the North American Chapter of the Association for Computational Linguistics

    work page 2019

  11. [11]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang

    work page

  12. [12]

    InProceed- ings of the International Conference on Machine Learning

    REALM: Retrieval-Augmented Language Model Pre-Training. InProceed- ings of the International Conference on Machine Learning

    work page

  13. [13]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Jo- hannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George B. M. van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laure...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  14. [14]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-Efficient Transfer Learning for NLP. InProceedings of the International Conference on Machine Learning

    work page 2019

  15. [15]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InProceedings of the International Conference on Learning Representations

    work page 2022

  16. [16]

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot Learning with Retrieval Augmented Language Models.Journal of Machine Learning Research24, 251 (2023), 1–43

    work page 2023

  17. [17]

    Herv’e J’egou, Matthijs Douze, and Cordelia Schmid. 2011. Product Quantization for Nearest Neighbor Search.IEEE Transactions on Pattern Analysis and Machine Intelligence33, 1 (2011), 117–128

    work page 2011

  18. [18]

    Jeff Johnson, Matthijs Douze, and Herv’e J’egou. 2019. Billion-Scale Similarity Search with GPUs.IEEE Transactions on Big Data7, 3 (2019), 535–547

    work page 2019

  19. [19]

    Scaling Laws for Neural Language Models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models.arXiv preprint arXiv:2001.08361(2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  20. [20]

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through Memorization: Nearest Neighbor Language Models. InProceedings of the International Conference on Learning Representations

    work page 2020

  21. [21]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the ACM Symposium on Operating Systems Principles

    work page 2023

  22. [22]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. InProceedings of the Conference on Empirical Methods in Natural Language Processing

    work page 2021

  23. [23]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. InProceedings of the International Con- ference on Machine Learning

    work page 2023

  24. [24]

    uttler, Mike Lewis, Wen-tau Yih, Tim Rockt

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K"uttler, Mike Lewis, Wen-tau Yih, Tim Rockt"aschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in Neural Informa- tion Processing Systems

    work page 2020

  25. [25]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. InProceedings of the Annual Meeting of the Association for Computational Linguistics

    work page 2021

  26. [26]

    Yu. A. Malkov and D. A. Yashunin. 2020. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence42, 4 (2020), 824– 836

    work page 2020

  27. [27]

    Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

    work page 2021

  28. [28]

    Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher R’e, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. InProceedings of the International Conference on Machine Learning

    work page 2023

  29. [29]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth’ee Lacroix, Baptiste Rozi‘ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models.arXiv preprint arXiv:2307.09288(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  31. [31]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. InAdvances in Neural Information Processing Systems

    work page 2017

  32. [32]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  33. [33]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. InProceedings of the USENIX Symposium on Operating Systems Design and Implementation

    work page 2022

  34. [34]

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou

    work page

  35. [35]

    Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models.arXiv preprint arXiv:2506.05176(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025