Pith. sign in

REVIEW 3 major objections 4 minor 43 references

Scaling a separate pretrained memory is more parameter-efficient than scaling the language model itself: a 410M-parameter backbone paired with a 6.9B memory outperforms a frozen 12B model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:01 UTC pith:PWBPPCPE

load-bearing objection Scaling study is real; domain results are threatened by training/eval overlap and the headline margin is razor-thin. the 3 major comments →

arxiv 2607.27919 v1 pith:PWBPPCPE submitted 2026-07-30 cs.CL

Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

Rubin Wei , Jiaqi Cao , Jiarui Wang , Junming Zhang , Qipeng Guo , Bowen Zhou , Zhouhan Lin This is my paper
classification cs.CL
keywords parametric long-term memorymemory decoderkNN distribution distillationparameter-efficient scalingdomain adaptationdistributed nearest-neighbor searchknowledge-intensive QAtraining-data memorization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that long-term memory need not be entangled with reasoning inside one set of decoder weights: memory can be pretrained standalone and scaled independently of the language model. Its central result is that, for a fixed parameter budget, spending more parameters on a pretrained memory and fewer on the backbone beats scaling the backbone alone — a frozen 410M-parameter model with a 6.9B memory averages 37.34 across 17 benchmarks, surpassing the 37.24 of a frozen 12B model with 39% fewer total parameters. The same recipe lifts domain performance: 1.7B memories for biology, law, and finance raise every backbone from 0.6B to 14B by more than 9 points on average, and the memories transfer across model families and vocabularies with a fraction of the training budget. Making this feasible at 207B tokens required solving a retrieval bottleneck with compressed, sharded, parallel nearest-neighbor search and sparse storage of the retrieval distributions. If these claims hold, improving a language model's knowledge can be bought as a swappable memory module rather than as a larger model.

Core claim

The paper claims that independently scaling a pretrained parametric memory is a more parameter-efficient route to better language-model performance than scaling the decoder-only backbone. The memory is a transformer trained to imitate the next-token distribution of a k-nearest-neighbor retriever over the corpus, then linearly interpolated with the frozen backbone's distribution at inference. Scaling to 300B tokens and memories up to 6.9B parameters, the paper reports gains on 47 of 51 task-scale combinations, concentrated in knowledge-heavy benchmarks, with a 410M backbone plus 6.9B memory surpassing a 12B model at 39% fewer parameters. Domain memories of 1.7B parameters improve biology, law

What carries the argument

The carrying mechanism is the memory decoder: a transformer initialized from a standard checkpoint and pretrained to reproduce a k-nearest-neighbor retriever's next-token distribution over the corpus, under a loss mixing Kullback-Leibler alignment with ordinary language modeling. The kNN distribution over plausible continuations carries richer supervision than the single observed token, so the memory absorbs corpus knowledge into weights that are linearly interpolated with the frozen backbone's distribution at inference. To make this tractable at 207B tokens, the paper compresses embeddings, shards the index across centroid ranges with parallel GPU search, and stores kNN distributions sparse

Load-bearing premise

The load-bearing premise behind the advertised domain gains is that the Biology-Instructions corpus used to train the domain memory does not overlap the Biology-Instructions benchmark used to evaluate it; the paper trains on that dataset after reformatting it as unsupervised text and evaluates on its benchmark with no deduplication analysis, and its own Section 6.6 probe shows the memory reproduces training suffixes far more often than a CPT baseline, so leakage could inflate

What would settle it

Search the Biology-Instructions training text against its benchmark evaluation splits for exact or near-duplicate examples, then re-score the smallest-backbone configuration (0.6B backbone plus 1.7B memory) on deduplicated splits and compare the average gain to the reported 9.88 points. If the gain collapses, the headline domain result is memorization rather than transferable memory; if it survives, the claim is confirmed. A complementary check: hold the interpolation coefficient fixed at one global value and verify that the Base + Memory configuration still beats the larger backbone at matche

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • At matched average scores, Base + Memory configurations use 33%, 32%, and 42% fewer total parameters than the 2.8B, 6.9B, and 12B backbone-only models.
  • The gains are broad, not outliers: 47 of 51 task-and-scale combinations improve, with the largest increases on knowledge-intensive benchmarks such as TriviaQA, 2WikiMultiHopQA, and HotpotQA.
  • Domain knowledge becomes a swappable component: a single 1.7B memory improves every tested backbone from 0.6B to 14B by more than 9 points on average across biology, law, and finance, with no continued pretraining of the backbone.
  • Memories generalize across model families: adapting a memory trained for one tokenizer to backbones with a different vocabulary, using 20% of the standard memory budget, improves the domain average by 4.26 and 7.77 points.
  • Memory makes training content more extractable: the memory model reproduces training suffixes more often than an attached continued-pretraining module (49.7% versus 42.4% exact continuation; 56.5% versus 22.6% on hidden anchor spans).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported advantage is measured with memories up to 6.9B parameters; the paper's curves do not show where memory scaling saturates. The natural test is a memory larger than 6.9B on the same 17 benchmarks — if average gains flatten, the parameter-efficiency claim has a ceiling.
  • A concentration effect follows from the memorization probe: if memory modules make training text easier to recover, the most extractable knowledge lives in the swappable module, which could be audited, replaced, or removed per deployment while the backbone stays fixed — a privacy property the paper does not develop.
  • The interpolation coefficient is tuned per task or domain when validation data exists; an open deployment question is how much of the gain survives with one globally fixed coefficient chosen without downstream labels.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper scales the Memory Decoder idea—a standalone parametric memory module trained to imitate kNN-retriever distributions—from the ~1B scale to 6.9B parameters and 300B training tokens. The engineering contribution is a distributed Faiss pipeline for constructing and storing kNN supervision over the 207B-token deduplicated Pile, together with sparse row storage that reduces memory overhead. The empirical claims are: (i) pairing a frozen Pythia-410M backbone with a 6.9B memory raises the 17-task average from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters; (ii) 1.7B domain memories improve Qwen3 backbones by more than 9 points on average across biology, law, and finance at every backbone scale; and (iii) domain memories transfer across vocabularies to OLMo models with 20% of the standard training budget. Additional analyses cover few-shot robustness, memory size, training budget, memorization extractability, and fidelity to kNN targets.

Significance. If the results hold, this is a significant contribution to modular language-model design: it would show that a pretrained parametric memory can be scaled independently of a frozen backbone and yield better parameter-performance tradeoffs than backbone scaling alone. The distributed indexing pipeline and sparse kNN-distribution storage are real engineering contributions that enable a previously impractical scale. The paper is also unusually transparent in reporting memorization probes (Section 6.6) and distribution-fidelity metrics (Section 6.7), which is commendable. However, the central empirical claims are currently not established because the evaluation setup does not rule out training/evaluation overlap, and the headline general-memory result rests on a 0.10-point average difference without uncertainty quantification. The paper's own memorization analysis makes the overlap concern concrete rather than hypothetical.

major comments (3)
  1. [§4 Datasets; §6.6; Table 2] The domain-memory evaluation has a direct train/eval-overlap risk that is not addressed. The biology memory is trained on Biology-Instructions after reformatting as unsupervised text and evaluated on the Biology-Instructions benchmark; law and finance use the same dataset family for training and evaluation. No deduplication, n-gram overlap analysis, or exclusion of benchmark examples from the training corpus is reported. This is load-bearing because Section 6.6 shows the memory model reproduces training suffixes much more often than a matched CPT model (49.7% vs 42.4% EM@8,16; 56.5% vs 22.6% on anchor completion). If a non-negligible fraction of benchmark examples appears in the training corpus, the advertised ">9 point" domain gains in Table 2 could be inflated by memorized retrieval continuations rather than by generalizable memory. The authors should either prove non-overlap or re-eva
  2. [§5.1; Table 1; §4 Implementation Details; Appendix F] The headline claim that 410M+Mem-6.9B surpasses Pythia-12B rests on a 0.10-point average difference (37.34 vs 37.24) across 17 tasks, with no error bars, confidence intervals, or significance test. The comparison is further complicated by the fact that the interpolation coefficient α is tuned per task on validation splits when available (Section 4, Table 15), and several α values are at the endpoints (0.00 or 1.00). Appendix F shows that improvements persist over broad α ranges for six tasks, but it does not provide variance estimates for the aggregate. The authors should report per-configuration variability (e.g., bootstrap over tasks/examples or seeds) and at least one fixed-α analysis to establish that the 0.10-point margin is not noise.
  3. [§3.1; §3.2; §9 Limitations] The parameter-efficiency comparison counts only the memory module's LM training budget (300B tokens) and parameter count, but the kNN-distribution construction requires an additional offline pass: generating hidden states with Pythia-6.9B over 207B tokens, indexing them, and performing one search per training query. Section 3.1 states that this is a quadratic-scale bottleneck that needed a distributed pipeline; Section 9 acknowledges the offline cost but does not quantify it. If the claim is strictly about parameter efficiency, this may be acceptable, but the text also says configurations are compared under 'matched total parameters and training budgets.' The preprocessing cost should be quantified or explicitly excluded from the efficiency claim to avoid overstating the practical advantage.
minor comments (4)
  1. [§4 Implementation Details] The interpolation coefficient α is said to be tuned on the validation split 'when validation data is available,' but Table 15 does not list a value for HaluEval. Please clarify how α was chosen for that benchmark and for any other task without a validation split.
  2. [Figure 1 and §5.2] The figure caption lists '33% fewer params / 32% fewer params / 42% fewer params' while the text reports the same percentages against the 2.8B, 6.9B, and 12B base models. The mapping is not immediately obvious; label the reference base model in each case.
  3. [Table 15] Several α values are equal to 0.00 (e.g., LogiQA at 2.8B/6.9B) or 1.00 (e.g., ARC-Challenge at 6.9B, NQ-Open at 2.8B). For α=0.00 the memory contributes nothing, and for α=1.00 the base model contributes nothing; these endpoints suggest the interpolation is doing more than a fixed blend. A short explanation of why these endpoints are selected would help.
  4. [§6.6] The memorization analysis is valuable, but the two probes use only 1,024 examples from BioInst. Reporting the 95% confidence interval for the EM differences (e.g., via bootstrap) would make the comparison more robust, especially since the difference on EM@8,16 (42.4% vs 49.7%) is based on a few hundred positive examples.

Circularity Check

0 steps flagged

No derivation-level circularity; central claims rest on external-benchmark experiments and controls, not on a fit or self-citation chain.

full rationale

This is an empirical scaling study. The central claims—that a 410M backbone plus a 6.9B memory surpasses Pythia-12B, and that domain memories give >9-point gains—are benchmark measurements against external tasks (MMLU, NQ-Open, TriviaQA, etc.) and baselines (Pythia, CPT, LoRA, RAG), not predictions derived from fitted values. The kNN-distillation objective (Eq. 1) and the interpolation rule (Eq. 2) are inherited from the authors' prior Memory Decoder work, but the paper does not justify its central claim by citing that work; it supplies new experiments, a matched CPT control (Table 5), and an attached-module control showing the memory objective rather than the interpolation interface drives gains. The only item that could be mistaken for circularity is the domain evaluation: §4 trains on Biology-Instructions and evaluates on the Biology-Instructions benchmark without reported deduplication, and §6.6 shows the memory can regurgitate training suffixes. That is a data-contamination / external-validity risk for the domain claims, not a circular derivation: a benchmark score is not the training objective by construction, the baselines are trained on the same corpora, and the general-memory claim is evaluated on 17 external benchmarks with Pile-trained baselines. No equation reduces a reported result to a fitted parameter or to an author-supplied theorem, so I find no significant circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim is an empirical scaling result. It rests on the memory objective, the kNN construction pipeline, and the evaluation protocol. The main fitted degrees of freedom are the per-task interpolation coefficient α and the unstated kNN hyperparameters K and τ. The method itself is inherited from self-cited prior work; no new physical or architectural entity is introduced beyond scaling an existing module.

free parameters (5)
  • interpolation coefficient α = per task, range 0.00–1.00 (Table 15)
    Eq. 2 mixes base and memory distributions; α is tuned on validation for each benchmark and drives the reported scores.
  • kNN temperature τ = not reported
    Eq. 4 weights retrieved neighbors via exp(-d/τ); the value is not stated though it directly changes the p_ret supervision signal.
  • neighbor count K = not reported
    The number of nearest neighbors used to form p_ret in Eq. 5 is not specified in the main text or appendices.
  • memory loss weight β = 0.5
    Eq. 1 balances KL(p_ret || p_ψ) and next-token CE; Appendix B fixes β = 0.5 for all runs.
  • sparse retention threshold ε = not quantified
    Eq. 6 removes probabilities below a 'small threshold'; the paper reports 64.95 retained pairs per row on the Pile but never gives ε.
axioms (5)
  • domain assumption A parametric model trained with Eq. 1 (KL + next-token CE) can internalize kNN-retrieval distributions without online retrieval.
    Central mechanism inherited from Cao et al. 2026; this paper scales it rather than proving it.
  • domain assumption kNN distributions over the Pile and domain corpora contain useful long-term knowledge beyond the ordinary next-token objective.
    Motivates p_ret as the training signal; if false, memory merely distills the base LM's own predictions.
  • domain assumption Hidden states of Pythia-6.9B are a suitable key/query representation for memories used with backbones from 410M to 12B.
    All general-memory datastores are built from Pythia-6.9B; transfer across backbone sizes assumes this representation is not backbone-specific.
  • domain assumption Approximate Faiss search with OPQ compression, IVF shards, and HNSW routing preserves enough retrieval fidelity for useful p_ret supervision.
    The distributed pipeline trades recall for tractability; no recall/precision analysis is reported.
  • domain assumption Evaluation benchmarks are not substantially contained in the memory training corpora, so gains measure generalization rather than memorization.
    Explicitly threatened by BioInst (train on Biology-Instructions, evaluate on Biology-Instructions benchmark) and by the §6.6 memorization results.

pith-pipeline@v1.3.0-daily-deepseek · 44723 in / 15858 out tokens · 142299 ms · 2026-07-31T23:01:44.431029+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory." pith.science (2026). https://pith.science/paper/PWBPPCPE

@misc{pith2026260727919,
  author       = {Pith},
  title        = {Pith review of: Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PWBPPCPE}},
  note         = {Machine review of arXiv:2607.27919}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduces a parametric long-term memory module but only studies it at a relatively small scale. In this work, we present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. We address this bottleneck with a distributed pipeline for Faiss indexing and retrieval, together with sparse, batch-wise loading of kNN distributions. Across model scales, we find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. On 17 benchmarks, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across the three domains by more than 9 points at every scale. Overall, our results demonstrate that independently scaling pretrained memory offers a more parameter efficient path to improving language model performance.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 1 canonical work pages

  1. [2]

    Bold marks the best score for each backbone, and underlining marks the second best. Model Base CPT LoRA RAG +Mem-0.6B +Mem-1.7B Qwen3 Family Qwen3-0.6B-Base17.19 21.09 22.37 18.79 24.87 28.48 Qwen3-1.7B-Base29.45 29.52 25.97 33.25 32.2034.58 Qwen3-4B-Base37.22 34.23 33.81 40.77 39.5641.64 Qwen3-8B-Base42.50 42.71 37.05 43.92 44.13 45.70 Qwen3-14B-Base43.6...

    arXiv 2034

  2. [4]

    Output: number EC4.3.3

    Prefix... Output: number EC4.3.3. Gold7,EC4.3.3.- identifies the enzyme’s function within it ×CPT model7,EC4.3.3.-4 indicates the presence of protein modifications ✓memory model7,EC4.3.3.- identifies the enzyme’s function within it Figure 17| Four additional FunctionEC continuations solved only by the memory model. All target suffixes occur in one BioInst...

    1913

  3. [8]

    The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    Pith/arXiv arXiv

  4. [10]

    Table 9|Implementation details for general memories. Configuration / Hyperparameter Memdec-1.4B Memdec-2.8B Memdec-6.9B Model configuration Architecture GPT-NeoX GPT-NeoX GPT-NeoX Hidden size 2048 2560 4096 Intermediate size 8192 10240 16384 Number of hidden layers 24 32 32 Number of attention heads 16 32 32 Activation function GELU GELU GELU Vocabulary s...

    2048

  5. [11]

    Biology-instructions: A dataset and benchmark for multi-omics sequence understanding capability of large language models.arXiv preprint arXiv:2412.19191,

    Haonan He, Yuchen Ren, Yining Tang, Ziyang Xu, Junxian Li, Minghao Yang, Di Zhang, Dong Yuan, Tao Chen, Shufei Zhang, et al. Biology-instructions: A dataset and benchmark for multi-omics sequence understanding capability of large language models.arXiv preprint arXiv:2412.19191,

    arXiv

  6. [12]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Pith/arXiv arXiv 2009

  7. [14]

    Generalization through memorization: Nearest neighbor language models.arXiv preprint arXiv:1911.00172,

    19 Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models.arXiv preprint arXiv:1911.00172,

    Pith/arXiv arXiv 1911

  8. [15]

    Halueval: A large-scale halluci- nation evaluation benchmark for large language models

    Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Halueval: A large-scale halluci- nation evaluation benchmark for large language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 6449–6464,

    2023

  9. [16]

    Logiqa: A challenge datasetformachinereadingcomprehensionwithlogicalreasoning.arXivpreprintarXiv:2007.08124,

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge datasetformachinereadingcomprehensionwithlogicalreasoning.arXivpreprintarXiv:2007.08124,

    Pith/arXiv arXiv 2007

  10. [17]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    Pith/arXiv arXiv

  11. [19]

    Olmo 3.arXiv preprint arXiv:2512.13961,

    Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, et al. Olmo 3.arXiv preprint arXiv:2512.13961,

    Pith/arXiv arXiv

  12. [21]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022,

    Pith/arXiv arXiv

  13. [23]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267,

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaugh- lin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267,

    Pith/arXiv arXiv

  14. [24]

    Memoryandbrainsystems: 1969–2009.JournalofNeuroscience,29(41):12711–12716,

    LarryRSquire. Memoryandbrainsystems: 1969–2009.JournalofNeuroscience,29(41):12711–12716,

    1969

  15. [25]

    Qwen Team. Qwen3. 5-omni technical report.arXiv preprint arXiv:2604.15804,

    Pith/arXiv arXiv

  16. [27]

    Infllm: Training-free long-context extrapolation for llms with an efficient context memory.Advances in neural information processing systems, 37:119638–119661, 2024a

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-free long-context extrapolation for llms with an efficient context memory.Advances in neural information processing systems, 37:119638–119661, 2024a. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient s...

    arXiv 2024

  17. [29]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–2380,

    2018

  18. [30]

    Vismem: Latent vision memory unlocks potential of vision-language models.arXiv preprint arXiv:2511.11007,

    21 Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory Xinlei Yu, Chengming Xu, Guibin Zhang, Zhangquan Chen, Yudong Zhang, Yongbo He, Peng-Tao Jiang, Jiangning Zhang, Xiaobin Hu, and Shuicheng Yan. Vismem: Latent vision memory unlocks potential of vision-language models.arXiv preprint arXiv:2511.11007,

    arXiv

  19. [31]

    Disc-lawllm: Fine-tuning large language models for intelligent legal services.arXiv preprint arXiv:2309.11325,

    Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, et al. Disc-lawllm: Fine-tuning large language models for intelligent legal services.arXiv preprint arXiv:2309.11325,

    Pith/arXiv arXiv

  20. [32]

    Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763,

    Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763,

    Pith/arXiv arXiv

  21. [33]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176,

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176,

    Pith/arXiv arXiv

  22. [34]

    Pre-training limited memory language models with internal and external knowledge.arXiv preprint arXiv:2505.15962,

    Linxi Zhao, Sofian Zalouk, Christian K Belardi, Justin Lovelace, Jin Peng Zhou, Ryan Thomas Noonan, Dongyoung Go, Kilian Q Weinberger, Yoav Artzi, and Jennifer J Sun. Pre-training limited memory language models with internal and external knowledge.arXiv preprint arXiv:2505.15962,

    arXiv

  23. [35]

    Training language models with memory augmentation

    Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5657–5673,

    2022

  24. [38]

    Table 14| Top-3 versus top-5 retrieval for the RAG baseline

    Neither retrieval depth consistently dominates across backbones or domains. Table 14| Top-3 versus top-5 retrieval for the RAG baseline. Bold marks the better retrieval depth for each backbone and metric. Model RAG BioInst LawBench FinEval Avg Qwen3-0.6B-Base top-33.89 18.36 24.86 15.70 top-5 2.96 17.82 21.37 14.05 Qwen3-1.7B-Base top-3 7.3329.8136.60 24....

    arXiv 2000

  25. [39]

    Zaheer Khan, Yuvraj Singh and Marlon Samuels made their ODI debuts during the competition

    It was their first win in a major ICC tournament. Zaheer Khan, Yuvraj Singh and Marlon Samuels made their ODI debuts during the competition. 0 0.25 0.5 0.75 1 Memory share of gold label probability: pmem/(pbase + pmem) Figure 12|Additional case study on a factual question about the2000 ICC KnockOut Trophy. In 2011, Kent County Cricket Club competed in Div...

    2011

  26. [42]

    BioInst Table 21|Per-task domain memory results on BioInst for the Qwen3-0.6B-Base backbone

    J.1. BioInst Table 21|Per-task domain memory results on BioInst for the Qwen3-0.6B-Base backbone. Task Metric Base CPT LoRA RAG +Mem 0.6B +Mem 1.7B DNA cpd MCC 2.96 21.27 32.83 -0.1734.9531.68 DNA emp MCC 1.40 7.08 -4.28 -3.2210.197.06 DNA pd MCC -2.03 10.79 18.17 -3.12 29.9640.24 DNA tf h MCC 0.54 6.71 -1.07 -5.92 21.0621.45 DNA tf m MCC 1.15 2.75 8.51 -...

    arXiv 1918

  27. [100]

    AVG is the macro average over the 25 task scores

    EM denotes exact match after answer normalization, MCC denotes Matthews correlation coefficient over extracted labels, and ROUGE-1 denotes the unigram F-measure for generated text. AVG is the macro average over the 25 task scores. Table 35|Per-task domain memory results on FinEval for the Qwen3-0.6B-Base backbone. Task Metric Base CPT LoRA RAG +Mem 0.6B +...

    arXiv 2065

  28. [381]

    Lawbench: Benchmarking legal knowledge of large language models

    18 Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, et al. Lawbench: Benchmarking legal knowledge of large language models. InProceedings of the 2024 conference on empirical methods in natural language processing, p...

    2024

  29. [2009]

    URL https://doi.org/10.1371/journal.pcbi.100

    doi: 10.1371/journal.pcbi.1000381. URL https://doi.org/10.1371/journal.pcbi.100

    work page doi:10.1371/journal.pcbi.1000381

  30. [2011]

    Gimme Shelter

    No. 2 Squadron is a Royal Australian Air Force( RAAF) squadron that operates from RAAF Base Williamtown, near Newcastle, New South Wales. From its formation in 1916 as part of the Australian Flying Corps, it has flown a variety of aircraft types including fighters, bombers, and Airborne Early Warning& Control( AEW& C). During World War I, the squadron ope...

    1916

  31. [2013]

    Deepsieve: Information sieving via llm-as-a-knowledge-router

    Minghao Guo, Qingcheng Zeng, Xujiang Zhao, Yanchi Liu, Wenchao Yu, Mengnan Du, Haifeng Chen, and Wei Cheng. Deepsieve: Information sieving via llm-as-a-knowledge-router. InFindings of the Association for Computational Linguistics: EACL 2026,

    2026

  32. [2014]

    Mlp mem- ory: A retriever-pretrained memory for large language models.arXiv preprint arXiv:2508.01832,

    Rubin Wei, Jiaqi Cao, Jiarui Wang, Jushi Kai, Qipeng Guo, Bowen Zhou, and Zhouhan Lin. Mlp mem- ory: A retriever-pretrained memory for large language models.arXiv preprint arXiv:2508.01832,

    arXiv

  33. [2016]

    Measuring and narrowing the compositionality gap in language models

    20 Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711,

    2023

  34. [2017]

    Demystifying domain- adaptive post-training for financial llms

    Zixuan Ke, Yifei Ming, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. Demystifying domain- adaptive post-training for financial llms. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31021–31047,

    2025

  35. [2018]

    Transformer-xl: Attentive language models beyond a fixed-length context.arXiv preprint arXiv:1901.02860,

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context.arXiv preprint arXiv:1901.02860,

    Pith/arXiv arXiv 1901

  36. [2019]

    Longrope: Extending llm context window beyond 2 million tokens.arXiv preprint arXiv:2402.13753,

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens.arXiv preprint arXiv:2402.13753,

    Pith/arXiv arXiv

  37. [2020]

    Measuring memorization in language models via probabilistic extraction

    Jamie Hayes, Marika Swanberg, Harsh Chaudhari, Itay Yona, Ilia Shumailov, Milad Nasr, Christopher A Choquette-Choo, Katherine Lee, and A Feder Cooper. Measuring memorization in language models via probabilistic extraction. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human La...

    2025

  38. [2021]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053,

    Pith/arXiv arXiv 1909

  39. [2022]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  40. [2023]

    2 olmo 2 furious.arXiv preprint arXiv:2501.00656,

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious.arXiv preprint arXiv:2501.00656,

    Pith/arXiv arXiv

  41. [2024]

    Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin

    URLhttps://arxiv.org/abs/2501.00663. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model.Journal of machine learning research, 3(Feb):1137–1155,

    Pith/arXiv arXiv

  42. [2025]

    Twinvoice: A multi-dimensional benchmark towards digital twins via llm persona simulation.arXiv preprint arXiv:2510.25536,

    Bangde Du, Minghao Guo, Songming He, Ziyi Ye, Xi Zhu, Weihang Su, Shuqi Zhu, Yujia Zhou, Yongfeng Zhang, Qingyao Ai, et al. Twinvoice: A multi-dimensional benchmark towards digital twins via llm persona simulation.arXiv preprint arXiv:2510.25536,

    arXiv

  43. [2026]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv