{"id":"85906fec-4a89-40a2-bb7d-35dbfadacdc2","arxiv_id":"2608.12419","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"LoKiFormer reports 1.33x faster pretraining convergence on a 5B proxy and benchmark scores that beat far larger models, with a fine-tuning dataset that is built around the evaluation benchmarks.","lead":"This paper adds a small convolution before attention and a learned key-value memory to an LLM, reporting 1.33x faster pretraining convergence. The benchmark gains it reports on a 7B model are far larger than any published 7B model, and they appear tied to a fine-tuning dataset built from the same benchmark questions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tables 2-4 likely measure the benchmark-aligned SFT distribution rather than the LFA/KMM architecture, because the SFT corpus is retrieved from concepts distilled from MMLU/CMMLU/C-Eval/GAOKAO and balanced to those benchmarks.","rationale":"The reader's verdict of REJECT is supported. The most load-bearing flaw is not an internal inconsistency or a minor statistical issue; it is that the main downstream evidence is produced by an SFT dataset engineered from the evaluation benchmarks themselves. The pipeline in Section C.2 is explicit: benchmark questions are used to derive retrieval seeds, and subject proportions are balanced to MMLU/CMMLU, so the instruction-tuning distribution is a surrogate for the test distribution. Even with exact-text decontamination, the SFT samples are selected because they cover the same concepts as the test questions. The paper's Figure 2 and KMM visualizations are also based on MMLU domains, reinforcing that the model is being tuned toward these benchmarks. The convergence-speedup claim in Section 5.2 is independently weakened by being a single unseeded 5B run, but the benchmark contamination would alone invalidate the paper's claims of state-of-the-art performance. The proposed concrete test of an alternative neutral SFT corpus would directly separate the architecture's contribution from the dataset's benchmark alignment. Since the reader already identified this SFT issue as the weakest assumption and reached REJECT, no verdict adjustment is needed.","tokens_in":26597,"tokens_out":6526,"duration_ms":70131,"concrete_test":"Re-train or fine-tune LoKiFormer-7B with a matched-size SFT corpus built without any seed concepts from MMLU/CMMLU/C-Eval/GAOKAO and without subject balancing to those benchmarks, using otherwise identical Stage 1, 3, and 5 pipelines. Evaluate MMLU, CMMLU, and C-Eval on the resulting model. If the scores drop substantially relative to 91.5/93.4/92.8 (e.g., by more than 5 points on MMLU), Tables 2-4 are inflated by benchmark-aligned SFT and cannot be attributed to LFA/KMM.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The downstream superiority claim (Tables 2-4) rests on an SFT dataset constructed in Section C.2 to be topically and distributionally aligned with the evaluation benchmarks. Stage 2 builds seed queries by extracting 10,000 questions from MMLU, CMMLU, C-Eval, and GAOKAO, distilling their underlying knowledge concepts, and then retrieving raw segments with cosine similarity > 0.65 to those concepts. Stage 4 then balances the subject proportions to match MMLU and CMMLU. Stage 5 removes near-duplicate text, but it cannot remove the intended conceptual alignment: the SFT samples are deliberately drawn from the same knowledge distribution as the test sets. Consequently, high MMLU/CMMLU/C-Eval scores after SFT may reflect exposure to the evaluation distribution rather than the benefits of LFA and KMM. The paper's own defense, Table 8, shows that LoKiFormer outperforms MAP-Neo-7B when both are trained on this same SFT dataset, but that comparison does not establish that the SFT dataset is neutral; it only shows that LoKiFormer exploits the benchmark-aligned data better. The base-model results in Table 7 (e.g., 74.3 MMLU for LoKiFormer-7B-Base) are far below the SFT scores in Tables 2-4, so the +17-point jump is exactly where the benchmark-aligned SFT could be inflating results. The central claim 'with less than one-tenth of the activated parameters it achieves state-of-the-art' is not supported unless the SFT construction is shown not to be the source of the gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LoKiFormer, an LLM architecture that extends a DeepSeek-style MLA backbone with two modules: Local Fusion Attention (LFA), a causal group convolution applied before Q/K/V construction to add a local inductive bias, and a Knowledge Memory Module (KMM), a fixed-size parameterized key-value store queried via softmax attention to provide explicit global knowledge. The paper reports a 1.33x faster pretraining convergence (reaching the baseline's 10k-step validation PPL at 7.5k steps in a 5B, 10k-step ablation), state-of-the-art downstream results for a 7B model with 3B activated parameters on MMLU, CMMLU, and C-Eval, ablations across model sizes from 1B to 60B, and analyses of field-domain specialization, field editability, and attention entropy.","tokens_in":26964,"tokens_out":13478,"duration_ms":123638,"significance":"If the convergence results hold, the practical contribution is a meaningful pretraining efficiency gain: roughly a 30% step reduction to a fixed validation PPL with only about 2.6% per-step overhead, translating to roughly 1.30x wall-clock speedup. The paper also provides a controlled ablation study (kernel size, number of groups, number of knowledge fields) and an interpretability package (field-domain specialization, field-removal experiments, key-space cosine similarity), which are useful strengths. The KMM field-editing experiment in Figure 5 is a particularly valuable causal test of the memory organization. These strengths are offset by the downstream evaluation being confounded by a benchmark-aligned SFT dataset and by convergence evidence resting on a single run with a suspicious train/eval PPL equality; both issues need to be resolved before the headline claims can be accepted.","major_comments":[{"comment":"The SFT dataset used for the main evaluation is constructed by extracting 10,000 questions from MMLU, CMMLU, C-Eval, and GAOKAO, distilling their knowledge concepts, retrieving raw segments with cosine similarity > 0.65 to those concepts, and balancing subject proportions to match MMLU and CMMLU (Stages 2 and 4). Stage 5 removes near-duplicate text but does not remove the intended conceptual alignment, so the SFT samples are drawn from the same knowledge distribution as the test sets. As a result, the large gains in Tables 2-4 (e.g., MMLU 74.3 for the base model in Table 7 to 91.5 after SFT in Table 2) may reflect exposure to the evaluation distribution rather than the benefits of LFA and KMM. The defense in Table 8 shows only that LoKiFormer exploits this benchmark-aligned data better than MAP-Neo-7B; it does not establish that the SFT data is neutral. The claim that the model 'with less than one-tenth of the activated parameters ... achieves state-of-the-art performance in most domains' therefore requires a control with an SFT dataset that is not topically aligned with the evaluation benchmarks, or an otherwise convincing demonstration that the SFT construction is not the source of the gains.","section":"Section C.2, Tables 2-4"},{"comment":"The baseline reports training and evaluation PPL both equal to 31.82 at 10k steps, while the augmented models show the expected gap (e.g., LoKiFormer 28.50 train vs 29.08 eval). Since the validation set is described as a 1% random sample of the Common Crawl portions of the training corpus, an exact equality of train and eval PPL to two decimals is implausible and suggests either leakage between validation and training or a reporting error. The central 1.33x speedup claim should be supported by corrected curves and by multiple independent runs; a single 10k-step run without error bars is insufficient for a claim of this strength.","section":"Section 5.2, Figure 3"},{"comment":"The main training setup is described inconsistently. Section 5.1 states that the 7B and 13B models use a global batch size of 16,384 and a 2048 context for 134k steps (about 4.5T tokens), whereas Section C.3 states that 'all main pre-training experiments use a global batch size of 16,384 and run for a fixed duration of 10,000 steps, consuming hundreds of billions of tokens'. This contradiction prevents the reader from determining the actual training budget of the 7B model whose downstream results are reported in Tables 2-4.","section":"Section 5.1 vs Section C.3"},{"comment":"The claim of state-of-the-art performance in 'most domains' is not supported by the reported numbers. In Table 3, LoKiFormer-7B (3B activated) is below Llama-3.1-405B on ARC-C (96.2 vs 96.9), HumanEval (87.0 vs 89.0), and GSM8K (73.0 vs 96.8), and below Llama-3.1-70B on GSM8K (73.0 vs 95.1). In Table 4, both Claude-3.5-Sonnet and Gemini-1.5-Pro exceed LoKiFormer on HellaSwag, HumanEval, and GSM8K. The statement should be restricted to the language-understanding benchmarks (MMLU, CMMLU, C-Eval) on which the advantage is actually observed.","section":"Section 5.3, Table 3"},{"comment":"The convergence advantage is demonstrated only for a 5B model trained for 10k steps (~21B tokens), whereas the downstream evaluations use a 7B model; no training convergence curve is shown for the 7B model. Figure 4 reports final validation PPL as a function of model size on WikiText-103 without specifying the token budget for each size, so the apparent scaling benefit is confounded with training compute. The claim that the architecture scales reliably to 60B should be accompanied by per-size token budgets and, ideally, convergence curves for at least the 7B model.","section":"Section 5.4, Figures 3 and 4"}],"minor_comments":[{"comment":"There are typos: 'respctively' in Section 4.1 and 'Konwledge Block Fields' in the caption of Figure 2 (and repeated in Figure 7).","section":"Section 4.1 and Figure 2 caption"},{"comment":"The notation 'KMM (F×h×du)' in Table 5 is ambiguous: it is unclear whether F is the total number of fields or the number of fields per group, and how h enters the parameter count. Please clarify the correspondence with Eq. (5) and Eq. (6).","section":"Table 5"},{"comment":"The caption of Figure 4 says 'Validation perplexity on WikiText-103', but Section 5.1 describes a validation set sampled from the English Common Crawl portions of the Matrix Data Pile. Please clarify which validation set is used in Figure 4.","section":"Figure 4 vs Section 5.1"},{"comment":"The field-removal experiment is performed only in Layer 4; the conclusion that 'KMM organizes knowledge into domain-specialized slots' should be qualified to the investigated layer, since the supplementary visualizations show layer-dependence of the field-domain associations.","section":"Section 5.4, Figure 5"},{"comment":"The MoE baseline is cited as 'Dai et al., 2024' in Table 2 but the supplementary MoE description cites 'Liu et al., 2024a'; please unify these references to the same DeepSeekMoE work.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue for any future round will be the SFT construction in Section C.2: without a control using a neutral, non-benchmark-aligned SFT dataset, the downstream superiority claims in Tables 2-4 are not interpretable. The authors should also re-examine the baseline train/eval PPL equality in Figure 3 and the training-duration contradiction between Section 5.1 and Section C.3 before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me skip the throat-clearing. LoKiFormer is an LLM architecture that bolts two cheap modules onto a DeepSeek-style MLA block: a grouped causal convolution before attention (LFA) and a learned key-value memory with cross-attention (KMM). Both are established ingredients, but the specific combination and the claimed 1.33x pretraining speedup are new. The per-step cost is honestly measured: LFA adds 0.41% parameters, KMM 2.12%, throughput overhead 2.6%. The ablations on kernel size, groups, and field count are reasonable. The field-removal experiment is a nice attempt to show the memory is interpretable.\n\nThe soft spots are real. The speedup rests on one 5B run with no seeds, and the baseline's train and eval PPL are reported as exactly equal (31.82), which is either a typo or a sign the numbers are too clean. The main 7B run never shows its own convergence curve, so you cannot verify that the 1.33x transfers. The downstream claims are worse. The SFT set in C.2 is built by distilling concepts from MMLU, CMMLU, C-Eval, and GAOKAO, retrieving raw segments with cosine similarity > 0.65 to those concepts, and balancing subject proportions to those benchmarks. That is not benchmark leakage in the literal sense, but it aligns the SFT distribution with the test distribution. The defense in Tables 7-8 shows the base model is strong and that it beats MAP-Neo on the same SFT data, which says the architecture exploits the aligned data better, not that the data is neutral. An MMLU of 91.5 on a 7B is outside the published envelope; without code, data, or a commit hash, the burden is not met.\n\nThere's also an internal contradiction in the training budgets: main text says 134k steps for 7B/13B, supplementary C.3 says all main runs are 10k steps. That kind of slip makes you trust the rest of the experimental reporting less.\n\nWho is this for? Someone working on efficient pretraining architectures might still find the cheap LFA idea worth stealing, and the KMM visualization suggests a testable way to inspect memory. But as a paper, the central claims are not supported. I would send it to a serious referee—the architecture is cheap and the claims are checkable—but I would expect that referee to require seeds, convergence curves, a neutral SFT comparison, and a real code release. If that doesn't happen, reject.","headline":"A plausible architecture with an unverifiable headline result: the convergence speedup rests on a single run and the downstream gains are likely contaminated by an SFT set built from the benchmarks.","tokens_in":27543,"tokens_out":4460,"would_cite":false,"duration_ms":40888,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Augmenting a decoder with a convolutional local-fusion layer and an addressable key–value memory makes LLM pretraining converge 1.33× faster, reaching the same validation loss in 7.5K steps instead of 10K at 2.6% per-step overhead.","keywords":["LoKiFormer","large language model pretraining","local fusion attention","knowledge memory module","mixture-of-experts","locality inductive bias","parametric key-value memory","knowledge editing"],"falsifier":"Pretrain the paper's baseline (same corpus, same 10K-step schedule, same MoE, no LFA/KMM), fine-tune it on the same 10B-token instruction set, and measure MMLU; if that model approaches the reported 91.5 instead of lagging far behind, the headline margins are mostly inherited from the fine-tuning distribution rather than the new modules. A complementary check is to repeat the 7.5K-versus-10K perplexity comparison on a different pretraining corpus and see whether the 1.33× step speedup persists.","tokens_in":26354,"feed_emoji":"⚡","tokens_out":20554,"duration_ms":172937,"temperature":0.7,"pith_summary":"LoKiFormer argues that LLM decoders are wasteful at two information scales: full self-attention redundantly re-learns short-range local structure, and mixture-of-experts buries reusable global knowledge inside expert weights where it can only be reached through computation. The proposed fix is two small, end-to-end-trainable modules: a causal group convolution that fuses neighboring tokens before attention (Local Fusion Attention, LFA), and a parametric key–value memory with addressable slots that tokens query directly (Knowledge Memory Module, KMM). On a 7-billion-parameter MoE model, the two together reach the baseline's 10K-step validation perplexity in 7.5K steps — 1.33× faster convergence — while the measured per-step throughput overhead is about 2.6%. If the claim holds, a model of given quality costs roughly a quarter less pretraining compute with no change to data, scale, or the attention's receptive field, which is the practical reason to care.","feed_headline":"LoKiFormer's two modules cut LLM pretraining steps by 25%","feed_subtitle":"Local-fusion convolution plus addressable key–value memory hits the target loss 1.33x sooner, at 2.6% per-step cost.","key_machinery":"The load-bearing mechanism is the pairing of two parameter-light, fully differentiable modules inside every decoder block. LFA is a causal 1D group convolution with kernel size $k=4$ and $h$ groups equal to the number of attention heads, applied to the token hidden states before the MLA down-projection: each output position aggregates its $k$ previous neighbors within its group (Eqn. 3), giving attention an explicit locality prior without shrinking its receptive field. KMM is a fixed set of learnable key–value memory fields with $F=64$ slots, queried by $H = c_{KV} W_H$, the projected MLA latent, through grouped scaled dot-product retrieval $Z^{(i)} = \\mathrm{softmax}(H^{(i)} K^{(i)\\top} / \\sqrt{d_u}) V^{(i)}$; the concatenated retrieval is projected and added to the MLA output before the MoE layer. The argument runs through the separation: local pattern fusion happens in the convolution, global knowledge lives in addressable slots, and attention and MoE are left to do what they are good at, with the specialization of fields by domain and the domain-specific effects of field deletion offered as evidence that the separation is real.","core_discovery":"The paper's central claim is that pretraining inefficiency is a decoupling failure, and that both halves can be fixed with minimal parameters. Local Fusion Attention puts an explicit locality bias in front of attention: a causal 1D group convolution with kernel size 4, grouped to match the attention heads, is applied to the hidden states before the multi-head latent attention down-projection, so the convolution absorbs short-range pattern fusion (0.41% of the 7B model's parameters, per the parameter breakdown) and attention is left to model broader context. Knowledge Memory Module stores global knowledge in $F=64$ learnable key–value fields, $K \\in \\mathbb{R}^{F \\times d_u}$ and $V \\in \\mathbb{R}^{F \\times d_v}$, queried by a projection of the MLA latent state through grouped scaled dot-product retrieval, with the retrieved vector added to the attention output before the MoE layer (2.12% of parameters). Trained together on a 4.5T-token bilingual corpus, the modules reach evaluation perplexity 29.08 at 10K steps versus 31.82 for the baseline; conversely, LoKiFormer hits the baseline's 10K-step perplexity at step 7.5K, a 1.33× speedup. The memory fields self-organize by domain with no supervision, and deleting a single field drops its own domain's score by up to 31.8%, which the paper reads as evidence that knowledge is stored in genuinely addressable form.","pith_inferences":["A test the paper leaves open: zeroing all KMM fields at inference should degrade knowledge-heavy tasks far more than syntactic or reasoning tasks, and the size of that gap would measure how much work the explicit memory does beyond the MoE layers it supplements.","The reported near-orthogonality of the 64 fields (maximum off-diagonal cosine ≈ 0.05) may not survive larger slot counts; since the paper's own ablations show perplexity still improving at $F=128$, checking whether specialization and clean removal effects persist at $F=128$ and beyond would show whether the interpretability is a small-$F$ phenomenon.","The 2.6% per-step overhead is measured at 4K context; re-measuring the convergence speedup at 32K–128K contexts, where convolution cost is fixed per token while memory retrieval grows with field count, would show whether the wall-clock advantage is context-length-dependent.","The strongest benchmark tables all follow supervised fine-tuning on a pool selected to resemble the knowledge benchmarks; a cleaner architecture-only split would pretrain the identical baseline without the two modules, fine-tune it on the same pool, and compare — the paper's same-SFT control uses a differently pretrained model, so the modules-versus-data split is not yet fully isolated."],"forward_implications":["Reaching a fixed validation loss takes about 25% fewer steps (7.5K instead of 10K), and with the measured 2.6% per-step throughput loss the total wall-clock pretraining cost drops by roughly a quarter.","The two modules are complementary: LFA alone gives 1.11× faster convergence and lifts base-model MMLU from 17.9 to 21.2, while adding KMM on top reaches 1.33× and 25.7, so neither module is redundant.","Knowledge becomes editable at the slot level: removing Field 32 drops algebra by 31.8% while leaving unrelated domains nearly intact, implying that facts can in principle be removed, updated, or audited without retraining.","The gains persist across scale: validation perplexity falls smoothly from 10.05 at 1B to 5.82 at 60B parameters, the 60B base model reaches 84.3 on MMLU, and training shows no divergence.","With about 3B activated parameters, the 7B model reports higher MMLU, CMMLU, and C-Eval scores than every listed baseline, including frontier models an order of magnitude larger; the paper attributes this to better local and global information integration during pretraining."],"supporting_citations":[{"why":"Supplies the multi-head latent attention backbone that LFA extends, and whose latent state $c_{KV}$ feeds the KMM queries.","marker":"Liu et al., 2024b"},{"why":"Defines the mixture-of-experts paradigm whose coupling of knowledge storage with computation KMM is designed to undo.","marker":"Shazeer et al., 2017"},{"why":"Provides the self-attention mechanism that LFA corrects by adding an explicit locality bias.","marker":"Vaswani et al., 2017"},{"why":"MMLU; supplies the five domains used for the field–domain association and field-removal analyses.","marker":"Hendrycks et al., 2021"},{"why":"Matrix Data Pile, the 4.5T-token bilingual corpus on which all LoKiFormer models are pretrained and validated.","marker":"Zhang et al., 2024"},{"why":"DeepSeekMoE; the shared-expert plus sigmoid-gated routed-expert formulation that LoKiFormer's MoE layer follows, cited in the supplementary.","marker":"Dai et al., 2024"}],"fun_headline_variants":["LoKiFormer's two fixes speed LLM pretraining 1.33x","25% fewer LLM pretrain steps via decoupled memory","Local fusion + explicit memory: LLMs train 1.33x faster","Addressable knowledge memory cuts LLM pretraining to 7.5K steps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the supervised fine-tuning data, built by retrieving raw text that is semantically similar to concepts distilled from MMLU, CMMLU, C-Eval, and GAOKAO and then balancing subject counts to match those benchmarks, gives the model no hidden advantage on those same benchmarks, so the large score gaps in the comparison tables come from the architecture rather than from the data.","fun_headline_variants_meta":{"raw":{"variants":["LoKiFormer's two fixes speed LLM pretraining 1.33x","25% fewer LLM pretrain steps via decoupled memory","Local fusion + explicit memory: LLMs train 1.33x faster","Addressable knowledge memory cuts LLM pretraining to 7.5K steps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000361,"raw_usage":{"total_tokens":2021,"prompt_tokens":1084,"completion_tokens":937,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":855}},"tokens_in":700,"tokens_out":937,"duration_ms":7716,"temperature":1.0,"reasoning_tokens":855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:29:03.889588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pretrain the paper's baseline (same corpus, same 10K-step schedule, same MoE, no LFA/KMM), fine-tune it on the same 10B-token instruction set, and measure MMLU; if that model approaches the reported 91.5 instead of lagging far behind, the headline margins are mostly inherited from the fine-tuning distribution rather than the new modules. A complementary check is to repeat the 7.5K-versus-10K perplexity comparison on a different pretraining corpus and see whether the 1.33× step speedup persists.","supporting_citations":[],"review_version":1}