Pith. sign in

REVIEW 5 major objections 5 minor 31 references

LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Augmenting a decoder with a convolutional local-fusion layer and an addressable key–value memory makes LLM pretraining converge 1.33× faster, reaching the same validation loss in 7.5K steps instead of 10K at 2.6% per-step overhead.

desk verdict A plausible architecture with an unverifiable headline result: the convergence speedup rests on a single run and the downstream gains are likely contaminated by an SFT set built from the benchmarks. read the letter →

arxiv 2608.12419 v1 pith:LHWNQN3F submitted 2026-08-12 cs.LG

Qiuwu Chen , Zimo Liu , Yuchen Li , Ying Sun , Yifan Zhang , Zhijie Qiu , Zeng You , Ryan Dong
show 3 more authors
Simeng Ma Yaofo Chen Mingkui Tan
This is my paper · ORCID
classification cs.LG
keywords LoKiFormerlargelanguagemodelpretraininglocalfusionattentionknowledgememorymodulemixture-of-expertslocalityinductivebiasparametrickey-valueediting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LoKiFormer argues that LLM decoders are wasteful at two information scales: full self-attention redundantly re-learns short-range local structure, and mixture-of-experts buries reusable global knowledge inside expert weights where it can only be reached through computation. The proposed fix is two small, end-to-end-trainable modules: a causal group convolution that fuses neighboring tokens before attention (Local Fusion Attention, LFA), and a parametric key–value memory with addressable slots that tokens query directly (Knowledge Memory Module, KMM). On a 7-billion-parameter MoE model, the two together reach the baseline's 10K-step validation perplexity in 7.5K steps — 1.33× faster convergence — while the measured per-step throughput overhead is about 2.6%. If the claim holds, a model of given quality costs roughly a quarter less pretraining compute with no change to data, scale, or the attention's receptive field, which is the practical reason to care.

What carries the argument

The load-bearing mechanism is the pairing of two parameter-light, fully differentiable modules inside every decoder block. LFA is a causal 1D group convolution with kernel size $k=4$ and $h$ groups equal to the number of attention heads, applied to the token hidden states before the MLA down-projection: each output position aggregates its $k$ previous neighbors within its group (Eqn. 3), giving attention an explicit locality prior without shrinking its receptive field. KMM is a fixed set of learnable key–value memory fields with $F=64$ slots, queried by $H = c_{KV} W_H$, the projected MLA latent, through grouped scaled dot-product retrieval $Z^{(i)} = \mathrm{softmax}(H^{(i)} K^{(i)\top} / \sqrt{d_u}) V^{(i)}$; the concatenated retrieval is projected and added to the MLA output before the MoE layer. The argument runs through the separation: local pattern fusion happens in the convolution, global knowledge lives in addressable slots, and attention and MoE are left to do what they are good at, with the specialization of fields by domain and the domain-specific effects of field deletion offered as evidence that the separation is real.

What would settle it

Pretrain the paper's baseline (same corpus, same 10K-step schedule, same MoE, no LFA/KMM), fine-tune it on the same 10B-token instruction set, and measure MMLU; if that model approaches the reported 91.5 instead of lagging far behind, the headline margins are mostly inherited from the fine-tuning distribution rather than the new modules. A complementary check is to repeat the 7.5K-versus-10K perplexity comparison on a different pretraining corpus and see whether the 1.33× step speedup persists.

Watch

Extended reading notes

Core claim

The paper's central claim is that pretraining inefficiency is a decoupling failure, and that both halves can be fixed with minimal parameters. Local Fusion Attention puts an explicit locality bias in front of attention: a causal 1D group convolution with kernel size 4, grouped to match the attention heads, is applied to the hidden states before the multi-head latent attention down-projection, so the convolution absorbs short-range pattern fusion (0.41% of the 7B model's parameters, per the parameter breakdown) and attention is left to model broader context. Knowledge Memory Module stores global knowledge in $F=64$ learnable key–value fields, $K \in \mathbb{R}^{F \times d_u}$ and $V \in \mathbb{R}^{F \times d_v}$, queried by a projection of the MLA latent state through grouped scaled dot-product retrieval, with the retrieved vector added to the attention output before the MoE layer (2.12% of parameters). Trained together on a 4.5T-token bilingual corpus, the modules reach evaluation perplexity 29.08 at 10K steps versus 31.82 for the baseline; conversely, LoKiFormer hits the baseline's 10K-step perplexity at step 7.5K, a 1.33× speedup. The memory fields self-organize by domain with no supervision, and deleting a single field drops its own domain's score by up to 31.8%, which the paper reads as evidence that knowledge is stored in genuinely addressable form.

Load-bearing premise

The load-bearing premise is that the supervised fine-tuning data, built by retrieving raw text that is semantically similar to concepts distilled from MMLU, CMMLU, C-Eval, and GAOKAO and then balancing subject counts to match those benchmarks, gives the model no hidden advantage on those same benchmarks, so the large score gaps in the comparison tables come from the architecture rather than from the data.

Editorial extensions

If this is right

  • Reaching a fixed validation loss takes about 25% fewer steps (7.5K instead of 10K), and with the measured 2.6% per-step throughput loss the total wall-clock pretraining cost drops by roughly a quarter.
  • The two modules are complementary: LFA alone gives 1.11× faster convergence and lifts base-model MMLU from 17.9 to 21.2, while adding KMM on top reaches 1.33× and 25.7, so neither module is redundant.
  • Knowledge becomes editable at the slot level: removing Field 32 drops algebra by 31.8% while leaving unrelated domains nearly intact, implying that facts can in principle be removed, updated, or audited without retraining.
  • The gains persist across scale: validation perplexity falls smoothly from 10.05 at 1B to 5.82 at 60B parameters, the 60B base model reaches 84.3 on MMLU, and training shows no divergence.
  • With about 3B activated parameters, the 7B model reports higher MMLU, CMMLU, and C-Eval scores than every listed baseline, including frontier models an order of magnitude larger; the paper attributes this to better local and global information integration during pretraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A test the paper leaves open: zeroing all KMM fields at inference should degrade knowledge-heavy tasks far more than syntactic or reasoning tasks, and the size of that gap would measure how much work the explicit memory does beyond the MoE layers it supplements.
  • The reported near-orthogonality of the 64 fields (maximum off-diagonal cosine ≈ 0.05) may not survive larger slot counts; since the paper's own ablations show perplexity still improving at $F=128$, checking whether specialization and clean removal effects persist at $F=128$ and beyond would show whether the interpretability is a small-$F$ phenomenon.
  • The 2.6% per-step overhead is measured at 4K context; re-measuring the convergence speedup at 32K–128K contexts, where convolution cost is fixed per token while memory retrieval grows with field count, would show whether the wall-clock advantage is context-length-dependent.
  • The strongest benchmark tables all follow supervised fine-tuning on a pool selected to resemble the knowledge benchmarks; a cleaner architecture-only split would pretrain the identical baseline without the two modules, fine-tune it on the same pool, and compare — the paper's same-SFT control uses a differently pretrained model, so the modules-versus-data split is not yet fully isolated.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes LoKiFormer, an LLM architecture that extends a DeepSeek-style MLA backbone with two modules: Local Fusion Attention (LFA), a causal group convolution applied before Q/K/V construction to add a local inductive bias, and a Knowledge Memory Module (KMM), a fixed-size parameterized key-value store queried via softmax attention to provide explicit global knowledge. The paper reports a 1.33x faster pretraining convergence (reaching the baseline's 10k-step validation PPL at 7.5k steps in a 5B, 10k-step ablation), state-of-the-art downstream results for a 7B model with 3B activated parameters on MMLU, CMMLU, and C-Eval, ablations across model sizes from 1B to 60B, and analyses of field-domain specialization, field editability, and attention entropy.

Significance. If the convergence results hold, the practical contribution is a meaningful pretraining efficiency gain: roughly a 30% step reduction to a fixed validation PPL with only about 2.6% per-step overhead, translating to roughly 1.30x wall-clock speedup. The paper also provides a controlled ablation study (kernel size, number of groups, number of knowledge fields) and an interpretability package (field-domain specialization, field-removal experiments, key-space cosine similarity), which are useful strengths. The KMM field-editing experiment in Figure 5 is a particularly valuable causal test of the memory organization. These strengths are offset by the downstream evaluation being confounded by a benchmark-aligned SFT dataset and by convergence evidence resting on a single run with a suspicious train/eval PPL equality; both issues need to be resolved before the headline claims can be accepted.

major comments (5)
  1. [Section C.2, Tables 2-4] The SFT dataset used for the main evaluation is constructed by extracting 10,000 questions from MMLU, CMMLU, C-Eval, and GAOKAO, distilling their knowledge concepts, retrieving raw segments with cosine similarity > 0.65 to those concepts, and balancing subject proportions to match MMLU and CMMLU (Stages 2 and 4). Stage 5 removes near-duplicate text but does not remove the intended conceptual alignment, so the SFT samples are drawn from the same knowledge distribution as the test sets. As a result, the large gains in Tables 2-4 (e.g., MMLU 74.3 for the base model in Table 7 to 91.5 after SFT in Table 2) may reflect exposure to the evaluation distribution rather than the benefits of LFA and KMM. The defense in Table 8 shows only that LoKiFormer exploits this benchmark-aligned data better than MAP-Neo-7B; it does not establish that the SFT data is neutral. The claim that the model 'with less than one-tenth of the activated parameters ... achieves state-of-the-art performance in most domains' therefore requires a control with an SFT dataset that is not topically aligned with the evaluation benchmarks, or an otherwise convincing demonstration that the SFT construction is not the source of the gains.
  2. [Section 5.2, Figure 3] The baseline reports training and evaluation PPL both equal to 31.82 at 10k steps, while the augmented models show the expected gap (e.g., LoKiFormer 28.50 train vs 29.08 eval). Since the validation set is described as a 1% random sample of the Common Crawl portions of the training corpus, an exact equality of train and eval PPL to two decimals is implausible and suggests either leakage between validation and training or a reporting error. The central 1.33x speedup claim should be supported by corrected curves and by multiple independent runs; a single 10k-step run without error bars is insufficient for a claim of this strength.
  3. [Section 5.1 vs Section C.3] The main training setup is described inconsistently. Section 5.1 states that the 7B and 13B models use a global batch size of 16,384 and a 2048 context for 134k steps (about 4.5T tokens), whereas Section C.3 states that 'all main pre-training experiments use a global batch size of 16,384 and run for a fixed duration of 10,000 steps, consuming hundreds of billions of tokens'. This contradiction prevents the reader from determining the actual training budget of the 7B model whose downstream results are reported in Tables 2-4.
  4. [Section 5.3, Table 3] The claim of state-of-the-art performance in 'most domains' is not supported by the reported numbers. In Table 3, LoKiFormer-7B (3B activated) is below Llama-3.1-405B on ARC-C (96.2 vs 96.9), HumanEval (87.0 vs 89.0), and GSM8K (73.0 vs 96.8), and below Llama-3.1-70B on GSM8K (73.0 vs 95.1). In Table 4, both Claude-3.5-Sonnet and Gemini-1.5-Pro exceed LoKiFormer on HellaSwag, HumanEval, and GSM8K. The statement should be restricted to the language-understanding benchmarks (MMLU, CMMLU, C-Eval) on which the advantage is actually observed.
  5. [Section 5.4, Figures 3 and 4] The convergence advantage is demonstrated only for a 5B model trained for 10k steps (~21B tokens), whereas the downstream evaluations use a 7B model; no training convergence curve is shown for the 7B model. Figure 4 reports final validation PPL as a function of model size on WikiText-103 without specifying the token budget for each size, so the apparent scaling benefit is confounded with training compute. The claim that the architecture scales reliably to 60B should be accompanied by per-size token budgets and, ideally, convergence curves for at least the 7B model.
minor comments (5)
  1. [Section 4.1 and Figure 2 caption] There are typos: 'respctively' in Section 4.1 and 'Konwledge Block Fields' in the caption of Figure 2 (and repeated in Figure 7).
  2. [Table 5] The notation 'KMM (F×h×du)' in Table 5 is ambiguous: it is unclear whether F is the total number of fields or the number of fields per group, and how h enters the parameter count. Please clarify the correspondence with Eq. (5) and Eq. (6).
  3. [Figure 4 vs Section 5.1] The caption of Figure 4 says 'Validation perplexity on WikiText-103', but Section 5.1 describes a validation set sampled from the English Common Crawl portions of the Matrix Data Pile. Please clarify which validation set is used in Figure 4.
  4. [Section 5.4, Figure 5] The field-removal experiment is performed only in Layer 4; the conclusion that 'KMM organizes knowledge into domain-specialized slots' should be qualified to the investigated layer, since the supplementary visualizations show layer-dependence of the field-domain associations.
  5. [References] The MoE baseline is cited as 'Dai et al., 2024' in Table 2 but the supplementary MoE description cites 'Liu et al., 2024a'; please unify these references to the same DeepSeekMoE work.

Circularity Check

1 steps flagged · score 4.0 of 10

Benchmark-aligned SFT construction makes Tables 2-4's benchmark-superiority claim partly circular; the 1.33x pretraining convergence result is not circular.

  1. fitted input called prediction [Section C.2 (SFT dataset construction), Stages 2 and 4; Section 5.3, Tables 2-4; Table 7 base-model results]
    "We first construct a knowledge seed query set by extracting 10,000 exam-style questions from MMLU, CMMLU, CEval, and GAOKAO, then distilling only their underlying knowledge concepts ... both seed queries and candidate samples are encoded using the bge-large-zh-v1.5 model, and top-k samples with cosine similarity > 0.65 are retained. ... We perform stratified sampling to balance the subject distribution according to the proportions in MMLU and CMMLU."

    Tables 2-4 report MMLU (91.5), CMMLU (93.4), and C-Eval (92.8) after SFT on a corpus selected (Stage 2) for cosine similarity > 0.65 to concepts distilled from exactly those benchmarks, and balanced (Stage 4) to MMLU/CMMLU subject proportions. The +17-point MMLU jump from the base model (74.3, Table 7) is thus attributable to the benchmark-aligned SFT distribution as much as to LFA/KMM. Table 8 only shows LoKiFormer uses this aligned SFT better than MAP-Neo; it does not show the SFT is neutral. The paper itself concedes the filter 'may inadvertently lead to data being overly specialized towards specific knowledge domains, potentially biasing the dataset.' Hence the 'state-of-the-art in most domains' claim is partially forced by construction.

full rationale

No equation-level circularity was found. The LFA and KMM equations (Eqs. 3 and 6) are constructive definitions, and the claimed 1.33x faster convergence to the same validation PPL (Section 5.2, Figure 3) is an empirical comparison under fixed settings that does not reduce to its inputs. The base-model gains in Table 7 and the same-SFT control in Table 8 provide independent evidence for the architecture, which is why the paper is not scored higher. However, the SFT corpus in Section C.2 is deliberately built to be topically and distributionally aligned with the MMLU/CMMLU/C-Eval/GAOKAO evaluation set: Stage 2 retrieves raw segments with cosine similarity > 0.65 to concepts distilled from those benchmarks, and Stage 4 balances subject proportions to match MMLU and CMMLU. Evaluating on those same benchmarks in Tables 2-4 therefore measures the constructed training distribution as well as the model; the decontamination step removes near-duplicate text but not the intended conceptual alignment. This makes the benchmark-superiority claim partially circular. Self-citations to Chen et al. (2025) and Zhang et al. (2025) are ordinary related-work attributions and are not load-bearing, so they do not affect the score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central convergence claim depends on three hyperparameters (k, g, F) that were selected by ablating the same 10k-step validation PPL reported as the result. The method also assumes the SFT data is not benchmark-shaped and that local inductive bias plus explicit memory improve pretraining. No formal axioms are introduced.

free parameters (4)
  • LFA kernel size k = 4
    Selected by ablation on the 5B validation PPL (Figure 6a); larger values not explored.
  • Number of convolution groups g in LFA = h (number of attention heads)
    Selected by ablation (Figure 6b); g=h reduces PPL versus g=1 and g=d.
  • Number of knowledge fields F in KMM = 64
    Selected by ablation (Figure 6c); F=128 gives lower PPL but F=64 chosen as a balance.
  • Number of KMM groups c = Not specified in main text
    The paper defines multi-group retrieval with c groups but never states c in the main text or algorithm; this is an unspecified free architectural choice.
assumptions (5)
  • domain assumption Natural language exhibits strong locality, so adjacent tokens are highly correlated at semantic levels (Section 4.2).
    This motivates LFA; if local fusion did not help, the convergence speedup would be a result of added parameters rather than inductive bias.
  • domain assumption KMM's fixed learned key-value fields can store reusable global knowledge and that retrieving from them helps pretraining (Section 4.3).
    The existence of domain-specialized fields is inferred from heatmaps, not from a mechanism; the claim that decoupled storage improves efficiency is assumed.
  • ad hoc to paper The SFT dataset constructed in Section C.2 is benchmark-clean and does not inflate scores on MMLU, CMMLU, and C-Eval (Stage 5).
    Decontamination relies on similarity thresholds and LLM-based structuring; the paper's own residual leakage estimate is below 0.1% but the retrieval-based construction is benchmark-shaped.
  • domain assumption Matrix Data Pile is a suitable pretraining corpus and the 1% cc-en sample approximates the validation distribution (Section 5.1).
    Standard domain assumption for empirical LLM papers.
  • standard math Standard math: softmax dot-product attention is differentiable and training converges with AdamW.
    Relying on standard transformer machinery, not proved in the paper.
invented entities (1)
  • Knowledge Memory Module fields (parameterized key-value slots K, V)
    purpose: Store global knowledge in addressable slots and retrieve via softmax dot-product
    No falsifiable handle outside the paper's own experiments; the claimed domain specialization is shown only through the authors' heatmaps and ablations. The component is an architectural contribution, not a physically postulated entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining." pith.science (2026). https://pith.science/paper/LHWNQN3F

@misc{pith2026260812419,
  author       = {Pith},
  title        = {Pith review of: LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LHWNQN3F}},
  note         = {Machine review of arXiv:2608.12419}
}
read the original abstract

Large language models (LLMs) have achieved remarkable breakthroughs across various applications. However, their architectures remain inefficient in pretraining due to two main limitations: (i) self-attention lacks an explicit inductive bias for locality, leading to redundant modeling of sequence-internal local information; (ii) mixture-of-experts (MoE) implicitly couples knowledge storage with computational pathways, hindering flexible access to sequence-external global knowledge. To overcome these limitations, we propose LoKiFormer, a novel LLM architecture that augments the standard decoder with two dedicated modules: 1) Local Fusion Attention (LFA), which incorporates a convolutional fusion to attention, explicitly capturing local patterns and allowing the attention to operate on more informative representations; 2) Knowledge Memory Module (KMM), which introduces a parametric key-value memory that explicitly stores global knowledge in addressable slots, decoupling storage from computation and enabling direct knowledge retrieval. Together, these modules enable LoKiFormer to achieve more efficient and effective integration of information at both levels. Experimental results show that LoKiFormer converges 1.33x faster in pre-training than baseline models, underscoring its superiority over existing LLM architectures.

Figures

Figures reproduced from arXiv: 2608.12419 by the authors.

Figure 1
Figure 1. Overall architecture of the decoder layer in our LoKiFormer. The model enhances a vanilla decoder block with two proposed modules: Local Fusion Attention (LFA) and the Knowledge Memory Module (KMM). Together with the Multi-Head Latent Attention (MLA) backbone and Mixture-of-Experts (MoE) layer, these components provide complementary local and global modeling capabilities, improving both modeling efficiency and knowl… view at source ↗
Figure 2
Figure 2. Field–domain associations of the proposed KMM with 64 knowledge fields, evaluated on five MMLU domains (1,024 samples each). The heatmap of softmax(HK/du) in layer 4 shows that fields emerge with domain-specific specialization, enabling explicit and interpretable knowledge retrieval. We put more visualizations of the remaining layers in Section D of the supplementary. as parameterized keys and values: K ∈ R F ×du , … view at source ↗
Figure 3
Figure 3. Effect of the proposed components LFA and KMM on pretraining performance. Left and right show the training perplexity curves and the evaluation perplexity on the validation set, respectively. All variants are pretrained for 10k steps under the same settings. 1B 5B 7B 13B 33B 60B Model Size 5.5 6.5 7.5 8.5 9.5 10.5 Perplexity (PPL) 10.05 9.59 9.40 8.86 7.43 5.82 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Validation perplexity on WikiText-103 across our LoKi￾Former models from 1B to 60B parameters. 5.3. Performance Comparisons Comparisons with Size-comparable Models. We compare LoKiFormer-7B with size-comparable open-source models across four domains in [PITH_FULL_IMAG…
Figure 5
Figure 5. Figure 5: Performance drop (%) on five MMLU domains after re￾moving individual knowledge fields in Layer 4. Each field primar￾ily affects its associated domain (diagonal entries), while exhibiting selective cross-domain impacts that reflect semantic relationships between discipl…
Figure 6
Figure 6. Figure 6: Ablations on the kernel size k and #convolutional group g in LFA, and #knowledge fields F. “BSL.” refers to “baseline”. Ablations of Kernel Sizes k in Local Fusion4 . We ab￾late the kernel size k of the convolution in the local fusion module with different k (i.e., k =…
Figure 7
Figure 7. Figure 7: Field–domain associations of the proposed KMM with 64 knowledge fields, evaluated on five MMLU domains (1,024 samples each). The heatmap of softmax(HK/du) shows that fields emerge with domain-specific specialization. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Per-layer average attention entropy difference between the model with and without Local Fusion Attention (LFA). Bars with blue borders indicate layers where LFA modules are inserted [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Per-layer average attention entropy of the model with Local Fusion Attention (LFA). 23 [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Per-layer average attention entropy of the model without Local Fusion Attention (LFA). 0 8 16 24 32 40 48 56 Knowledge Field Index 0 8 16 24 32 40 48 56 Knowledge Field Index layer 4 (Max |off-diag| = 0.0513) 0 8 16 24 32 40 48 56 Knowledge Field Index 0 8 16 24 32 40…
Figure 11
Figure 11. Figure 11: Cosine similarity matrix between 64 KMM knowledge fields across different layers. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 6 canonical work pages

  1. [3]

    Chen, J., Jian, P., Xi, T., Yi, D., Du, Q., Ding, C., Zhu, G., Zong, C., Wang, J., and Zhang, J

    Accessed: 2025-01-20. Chen, J., Jian, P., Xi, T., Yi, D., Du, Q., Ding, C., Zhu, G., Zong, C., Wang, J., and Zhang, J. Chinesewebtext: Large- scale high-quality chinese web text extracted with effec- tive evaluation model.arXiv preprint arXiv:2311.01149,

    arXiv 2025

  2. [7]

    Gen- erating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

    Child, R., Gray, S., Radford, A., and Sutskever, I. Gen- erating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

    arXiv 1904

  3. [10]

    Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models.arXiv preprint arXiv:2308.10755,

    He, C., Jin, Z., Xu, C., Qiu, J., Wang, B., Li, W., Yan, H., Wang, J., and Lin, D. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models.arXiv preprint arXiv:2308.10755,

    arXiv

  4. [12]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024a. Jiang, H., LI, Y ., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y ., Yang, Y ., and Qiu, L. MInference 1.0: Accelerating...

    arXiv

  5. [13]

    J., Greenlee, F., Thomas, G., Purtorab, M., and Toulis, A

    Kang, J., Wu, W., Christianos, F., Chan, A. J., Greenlee, F., Thomas, G., Purtorab, M., and Toulis, A. Lm2: Large memory models.arXiv preprint arXiv:2502.06049,

    arXiv

  6. [14]

    Cmmlu: Measuring massive multitask language understanding in chinese

    Li, H., Zhang, Y ., Koto, F., Yang, Y ., Zhao, H., Gong, Y ., Duan, N., and Baldwin, T. Cmmlu: Measuring massive multitask language understanding in chinese. InFindings of the Association for Computational Linguistics ACL 2024, pp. 11260–11285,

    work page 2024

  7. [15]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024a

    Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024a. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical repor...

    arXiv

  8. [16]

    Llm360: Towards fully transparent open-source llms

    Liu, Z., Qiao, A., Neiswanger, W., Wang, H., Tan, B., Tao, T., Li, J., Wang, Y ., Sun, S., Pangarkar, O., et al. Llm360: Towards fully transparent open-source llms. InFirst Conference on Language Modeling, 2024c. Luo, Y ., Kong, Q., Xu, N., Cao, J., Hao, B., Qu, B., Chen, B., Zhu, C., Zhao, C., Zhang, D., et al. Yayi 2: Multilin- gual open-source large la...

    arXiv

Show all 31 references
  1. [17]

    and Lin, S

    Mu, S. and Lin, S. A comprehensive survey of mixture- of-experts: Algorithms, theory, and applications.arXiv preprint arXiv:2503.07137,

  2. [18]

    D., Man, H., Ngo, N

    Nguyen, T., Van Nguyen, C., Lai, V . D., Man, H., Ngo, N. T., Dernoncourt, F., Rossi, R. A., and Nguyen, T. H. Cul- turax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. InProceedings 11 LoKiFormer: Locality-aware Attention with Decou...

    2024

  3. [19]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    OpenAI. GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  4. [20]

    G., Stoica, I., and Gonzalez, J

    Packer, C., Wooders, S., Lin, K., Fang, V ., Patil, S. G., Stoica, I., and Gonzalez, J. E. Memgpt: Towards llms as operating systems.arXiv preprint arXiv:2310.08560,

    arXiv

  5. [21]

    Dense training, sparse infer- ence: Rethinking training of mixture-of-experts language models.arXiv preprint arXiv:2404.05567,

    Pan, B., Shen, Y ., Liu, H., Mishra, M., Zhang, G., Oliva, A., Raffel, C., and Panda, R. Dense training, sparse infer- ence: Rethinking training of mixture-of-experts language models.arXiv preprint arXiv:2404.05567,

    arXiv

  6. [22]

    Hunyuan-large: An open-source moe model with 52 billion activated pa- rameters by tencent.arXiv preprint arXiv:2411.02265,

    Sun, X., Chen, Y ., Huang, Y ., Xie, R., Zhu, J., Zhang, K., Li, S., Yang, Z., Han, J., Shu, X., et al. Hunyuan-large: An open-source moe model with 52 billion activated pa- rameters by tencent.arXiv preprint arXiv:2411.02265,

    arXiv

  7. [23]

    I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al

    Team, G., Georgiev, P., Lei, V . I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024a. Team, G., Mesnard, T., Hardin...

    arXiv

  8. [24]

    Skywork: A more open bilingual foundation model.arXiv preprint arXiv:2310.19341,

    Wei, T., Zhao, L., Zhang, L., Zhu, B., Wang, L., Yang, H., Li, B., Cheng, C., L ¨u, W., Hu, R., et al. Skywork: A more open bilingual foundation model.arXiv preprint arXiv:2310.19341,

    arXiv

  9. [25]

    Skywork-moe: A deep dive into training techniques for mixture-of-experts language models.arXiv preprint arXiv:2406.06563,

    Wei, T., Zhu, B., Zhao, L., Cheng, C., Li, B., L¨u, W., Cheng, P., Zhang, J., Zhang, X., Zeng, L., et al. Skywork-moe: A deep dive into training techniques for mixture-of-experts language models.arXiv preprint arXiv:2406.06563,

    arXiv

  10. [26]

    Qwen2.5 technical report

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, ...

    arXiv

  11. [27]

    L., Pan, D., Cheng, E., Liu, J., Lin, Q., et al

    Zhang, G., Qu, S., Liu, J., Zhang, C., Lin, C., Yu, C. L., Pan, D., Cheng, E., Liu, J., Lin, Q., et al. Map-neo: Highly capable and transparent bilingual large language model series.arXiv preprint arXiv:2405.19327,

    arXiv

  12. [29]

    is a large-scale, 4.5 trillion-token bilingual pre-training corpus meticulously curated for the MAP-Neo model series. The English subset is derived from a re-processing of high-quality public datasets, including RedPajama-Data-V2 (Weber et al., 2024), Dolma (Soldaini et al., 2...

    2024

  13. [30]

    It contains 7,473 training and 1,319 hand-written test questions, each requiring two to eight sequential reasoning steps

    is a benchmark for assessing the multi-step mathematical reasoning capabilities of language models through elementary school-level word problems. It contains 7,473 training and 1,319 hand-written test questions, each requiring two to eight sequential reasoning steps. The probl...

    2021

  14. [31]

    Newton,” “Calculus,

    to verify whether the Knowledge Block spontaneously acquires structured, domain-specific representations. The primary objective was to observe if high-relevance tokens from distinct domains focus on different key regions via the cross-attention mechanism. Architecturally, a tr...

    2026

  15. [2018]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  16. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  17. [2020]

    Internlm2 technical report.arXiv preprint arXiv:2403.17297,

    Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., et al. Internlm2 technical report.arXiv preprint arXiv:2403.17297,

    arXiv

  18. [2021]

    Chatdb: Augmenting llms with databases as their sym- bolic memory.arXiv preprint arXiv:2306.03901,

    Hu, C., Fu, J., Du, C., Luo, S., Zhao, J., and Zhao, H. Chatdb: Augmenting llms with databases as their sym- bolic memory.arXiv preprint arXiv:2306.03901,

    arXiv

  19. [2022]

    We organize our supplementary as follows

    13 LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient LLM Pretraining SUPPLEMENTARYMATERIALS In the supplementary, we provide more details about our LoKiFormer architecture and more implementation details. We organize our supplementary as follow...

    2024

  20. [2023]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    arXiv

  21. [2024]

    anthropic.com/news/claude-3-5-sonnet

    URL https://www. anthropic.com/news/claude-3-5-sonnet. Beltagy, I., Peters, M. E., and Cohan, A. Long- former: The long-document transformer.arXiv preprint arXiv:2004.05150,

    2004 arXiv

  22. [2025]

    Conditional memory via scalable lookup: A new axis of sparsity for large language models.arXiv preprint arXiv:2601.07372,

    Cheng, X., Zeng, W., Dai, D., Chen, Q., Wang, B., Xie, Z., Huang, K., Yu, X., Hao, Z., Li, Y ., et al. Conditional memory via scalable lookup: A new axis of sparsity for large language models.arXiv preprint arXiv:2601.07372,

    arXiv

  23. [2026]

    Mem0: Building production-ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413,

    Chhikara, P., Khant, D., Aryan, S., Singh, T., and Yadav, D. Mem0: Building production-ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413,

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.