Moir: Let the Model Direct Its Own Story for Robust Cross-Domain Knowledge Editing

Dong-Kyum Kim; Jea Kwon; Jiwon Kim; Meeyoung Cha

REVIEW 4 major objections 5 minor 20 references

The paper argues that knowledge editing destroys math and code reasoning because the preservation covariance is built from the wrong distribution, and shows that sampling the model's own continuations seeded with one random token fixes it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:25 UTC pith:7OEGHI3E

load-bearing objection The core idea—estimating the preservation covariance from the model's own random-seeded generations—is genuinely new and worth studying, but the paper's 'consistent' preservation claim is contradicted by its own tables; the real effect is mostly Qwen3-AlphaEdit-specific. the 4 major comments →

arxiv 2607.20433 v1 pith:7OEGHI3E submitted 2026-05-10 cs.CL cs.AI

Moir: Let the Model Direct Its Own Story for Robust Cross-Domain Knowledge Editing

Jea Kwon , Jiwon Kim , Dong-kyum Kim , Meeyoung Cha This is my paper

classification cs.CL cs.AI

keywords knowledge editingmodel editingcovariance preservationself-generationcapability preservationMEMITAlphaEditdistributional shift

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that knowledge editing destroys math and code abilities because the covariance matrix used to protect existing knowledge is built from the wrong distribution—an encyclopedic text corpus that does not match what the model actually internalized after SFT and DPO. It proposes MOIR: build that covariance from the model's own sampled continuations, seeded by a single random vocabulary token. This change keeps GSM8K math accuracy at 79.9% after 20,000 batch edits where the encyclopedic-corpus baseline falls to 10.9%, across several 7–8B models and two editing methods. The broader claim is that the preservation space for covariance-based editors can be derived from the model itself, without any external data.

Core claim

MOIR's central assertion is that the self-generated covariance CMOIR samples from the model's actual internalized distribution, covering exactly the subspaces the model relies on, regardless of whether they appear in any external corpus. The paper proves that for closed-form covariance-constrained editors, the update depends on the input distribution only through the uncentered covariance C, because the output-side factor G cancels in the closed-form optimum (Proposition 1). Empirically, replacing the standard encyclopedic proxy with CMOIR delays or prevents the collapse of GSM8K and HumanEval under both MEMIT and AlphaEdit, in batch and sequential regimes, across OLMo-2, Llama-3.1, and Qwen

What carries the argument

The uncentered activation covariance C = (1/N) Σ k_i k_i^T, computed at the input to the targeted MLP layer, is the single distributional object that determines a covariance-constrained editor's update. MOIR constructs C from autoregressive continuations seeded with one uniformly random vocabulary token, which breaks the instruction-template attractor and lets the model sample the broader subspaces it has internalized. Proposition 1 shows that the output-side K-FAC factor G cancels in the closed-form optimum, so the choice of C—self-generated versus corpus-derived—fully controls what gets preserved.

Load-bearing premise

The load-bearing premise is that continuations seeded with one random token reproduce the model's true operative activation distribution closely enough that the estimated covariance covers the math and code subspaces; if the model's decoding is mode-collapsed or the random seed never enters those regions, self-generation misses them.

What would settle it

Run MOIR on a model whose decoding distribution is pathologically mode-collapsed, for example after extreme RLHF, then apply 20,000 AlphaEdit batch edits and measure GSM8K accuracy; if math accuracy collapses despite MOIR because the self-generated covariance under-samples that specialized domain, the core approximation fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Knowledge editing no longer requires access to pretraining or post-training corpora; a roughly two-hour one-time generation from the deployed model provides the preservation covariance.
Covariance-based editors can sustain tens of thousands of batch and sequential edits without collapsing math and code reasoning, as long as their preservation space is aligned with the model's operative distribution.
The model itself is the most accessible source of the shifted post-training manifold; static corpora, including the original pretraining mixture, are systematically biased proxies.
Since the editing update depends only on the input covariance, improving the distribution used to estimate C is a strict improvement across editors without architectural changes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the self-generated covariance truly tracks the operative distribution, the same idea could be used to audit which capability subspaces a deployed model has internalized, separate from any editing task.
A testable extension the paper leaves open is whether memory-based or hypernetwork editors, which use covariance-like operators, would also benefit from self-generated preservation spaces.
The single-random-token recipe suggests a general principle for probing internalized distributions: a minimal off-manifold perturbation escapes post-training attractors while staying within the model's learned statistics; this could inform data-free evaluation beyond editing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The core idea—estimating the preservation covariance from the model's own random-seeded generations—is genuinely new and worth studying, but the paper's 'consistent' preservation claim is contradicted by its own tables; the real effect is mostly Qwen3-AlphaEdit-specific.

read the letter

The novel piece here is real: sampling the preservation covariance C from the model's own random-token-seeded generations, rather than from Wikipedia or a pretraining mix, is a fresh and plausible move. Proposition 1 (G-independence) is correctly stated and proved, and the OLMo-Mix oracle ablation is a nice control. The NLL distributional analysis also supports the claim that post-training shifts the model's operative manifold away from static corpora. That part of the paper is solid and useful.

The soft spot is the empirical overclaim. The abstract and Section 6 say MOIR "consistently extends preservation" across editors, models, and regimes. The paper's own tables don't support that. In AlphaEdit batch, CMOIR gives essentially no gain on OLMo-2 and Llama-3.1 (Table 11: e.g., OLMo-2 5K .600 vs .603; Llama-3.1 5K .662 vs .669). In MEMIT sequential, CMOIR delays collapse on OLMo-2 and Llama-3 but still collapses to zero preservation HM by 500 edits (OLMo-2) and 200 edits (Llama-3) (Table 10), contradicting the "sustains 0.5–0.7" claim. The headline 79.9% GSM8K result is real but specific to Qwen3-8B with AlphaEdit. So the general principle is not established; the effect is model- and regime-dependent.

A few smaller issues: no confidence intervals anywhere, so some reversals may be noise. The seed-length choice (k=1) is tuned on the same harmonic-mean score used in the main results, which is a selection-on-evaluation concern. And the load-bearing approximation in Eq. 4—that a single random token makes self-generated activations proxy the training distribution—is an assumption the authors themselves hedge in Limitations. It may hold for these instruct models but is not shown to hold generally.

Who is this for? Researchers working on knowledge editing and model updating will want to know this idea. The method deserves further study, and a serious referee should see it. But the authors need to narrow their claims, add error bars, and report per-model/per-regime results without the "consistent" framing. The paper as written is not ready for acceptance, but it is not a desk reject either. I'd send it to review and ask for major revision.

Referee Report

4 major / 5 minor

Summary. The paper argues that the cross-domain capability collapse observed in covariance-based knowledge editors (MEMIT, AlphaEdit) stems from the choice of the preservation distribution used to estimate C. It proposes MOIR, which estimates C from the model's own decoding distribution by seeding generation with a single random vocabulary token. Claims are made that this 'self-generated manifold' consistently extends preservation across OLMo-2, Llama-3.1, and Qwen3, in both batch and sequential regimes, up to 20K edits, with a headline GSM8K recovery from 10.9% to 79.9% on Qwen3-8B under AlphaEdit batch editing. The paper also proves Proposition 1 (the parameter update depends on the input distribution only through the input covariance) and Lemma 3 (TV-Lipschitz continuity of the covariance map).

Significance. If the central empirical claim were true, the contribution would be significant: a data-free, drop-in replacement for the preservation covariance that consistently extends the usable edit budget across models and editors. The theoretical component is sound: Proposition 1 is correct and cleanly isolates the role of C, and Lemma 3 provides a usable bound. The method is also practical, requiring only a one-time generation pass. However, the paper's own tables contradict the 'consistently extends preservation' claim. The effect is large and clear only for Qwen3-8B under AlphaEdit; in several OLMo-2 and Llama-3 conditions, CMOIR is indistinguishable from or slightly worse than CWiki. Without confidence intervals or a pre-registered multi-seed analysis, the current evidence does not establish the proposed general principle, only a promising mechanism for a specific model/editor combination.

major comments (4)

[Section 6, Table 10] The claim that 'CMOIR sustains 0.5–0.7 preservation throughout 10^4 edits' under MEMIT sequential editing is directly contradicted by Table 10. OLMo-2: preservation HM is 0.503 at 200 edits, then 0.000 at 500, 1K, 2K, and 5K edits. Llama-3: preservation HM is 0.460 at 100 edits, then 0.000 at 200 edits and all later counts. Only Qwen3 retains non-zero preservation at 5K (0.657). Thus the 'across all three models' statement in Section 6 is false.
[Table 11 and Table 12] The 'consistently extends preservation' claim also fails under AlphaEdit. In Table 11 (batch), CMOIR is slightly worse than CWiki at OLMo-2 5K (0.600 vs 0.603) and Llama-3 5K (0.662 vs 0.669). In Table 12 (sequential), Llama-3 is consistently worse at 500 (0.670 vs 0.675), 1K (0.668 vs 0.676), and 2K (0.661 vs 0.672) edits. These reversals are small, but the paper claims consistency and reports no confidence intervals, so they cannot be dismissed as noise.
[Section 4, Table 2] The key design choice, seed length k=1, is selected on OLMo-2 using the same Se/Sp harmonic-mean scores that later serve as the headline evaluation metrics. This is selection-on-evaluation risk. The paper reports only a single run of each configuration and no variance across seeds or generated samples. A proper validation would hold out the seed-length choice or report bootstrap/interval estimates before claiming 'consistent' improvements elsewhere.
[Appendix A.1, Eq. (4)] The load-bearing approximation E_{x~pθ(·|s)}[φφ^T] ≈ E_{x~Dtrain}[φφ^T] is asserted rather than demonstrated. The NLL/JSD analyses in Figure 3 and Figure 11 are suggestive about token-level distributions, but they do not directly validate equality at the activation-covariance level. The paper's own Limitations section concedes that extreme RLHF mode collapse would under-sample rare capabilities. This is an additional correctness risk, though the empirical contradictions in Tables 10–12 are already sufficient to undermine the central claim.

minor comments (5)

[Section 3.1] The text says 'Please refer to Appendix C for the full mathematical formulations,' but the parameter-update formulations are in Appendix B, not Appendix C.
[Table 10 caption] The caption 'MEMIT sequential editing (cached)' uses 'cached' without explaining what is cached; either define it or remove it.
[Section 1, final paragraph] 'The tested models sustain 79.9% accuracy on GSM8K after 20,000 edits' is inaccurate: only Qwen3-8B under AlphaEdit batch achieves this, as the abstract itself states.
[Figure 5] The sentence 'Figure 5 and 13 shows task-wise preservation' mixes singular/plural; should read 'Figures 5 and 13 show.'
[Table 4] The row 'Qwen3-8B' likely denotes the instruct variant but is not labeled as such, unlike 'OLMo2-7B-Instruct' and 'Llama-3.1-8B-Instruct.'

Circularity Check

0 steps flagged

No significant circularity: CMOIR is a parameter-free sample covariance, Proposition 1 is proven in-appendix, and the headline results are external-benchmark measurements.

full rationale

The derivation chain is not circular. Proposition 1 (Sec. 5.1, App. B.2) is a self-contained proof that closed-form covariance-constrained updates depend only on the input covariance A; it does not assume the empirical preservation results. CMOIR (Eq. 3, Algorithm 1) is an uncentered sample covariance of keys extracted from the model's own unconditional generations; it contains no parameters fitted to preservation benchmarks. The load-bearing premise, Eq. 4, is explicitly stated as an approximation ('The approximation is exact when p_theta(.|s)=D_train marginally over s'), and the Limitations section concedes the extreme RLHF mode-collapse failure mode; this is an empirical assumption, not a reduction of the conclusion to the premise. The preservation evaluations (Tables 9-12) use external benchmarks that are not used to construct C, so the GSM8K result is an independent measurement, not a renamed input. The main caveat is that seed length k=1 is selected from Table 2 using the same Se/Sp harmonic means used in one evaluation arm, but this is a hyperparameter-selection risk, not a construction-level circularity, and it does not make the held-out Qwen3/Llama3 results forced by definition. No load-bearing self-citations or imported uniqueness claims appear.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new entities are postulated. The method relies on standard K-FAC factorization and the key ad hoc assumption that single-token-seeded self-generation captures the model's operative distribution. The free parameters are design choices and editing hyperparameters, not parameters fitted inside C itself.

free parameters (4)

Seed length k (number of random prefix tokens) = 1
Central design choice. Selected in Section 4, Table 2 by comparing total Edit HM + Pres. HM on the same evaluation metrics (Se, Sp) used for the main results. No held-out validation for this hyperparameter.
Number of generated sequences N (MOIR) = 100,000
Chosen to match the conventional Wikipedia sample size (100K articles). MOIR yields ~17–25M tokens vs ~54.8M tokens for Wikipedia; no ablation on token count.
MEMIT regularization weight lambda = 15,000
Selected from grid {0.5K, 1.5K, 5K, 15K, 30K}. The selection criterion and per-model breakdown are not fully reported.
AlphaEdit null-space projection threshold tau = Per-model from grid {0.02, 0.005, 0.001, 0.0005}
Selected per model on a held-out validation split; the actual chosen values are not listed in the paper.

axioms (4)

domain assumption K-FAC factorization F ≈ A ⊗ G holds for the targeted MLP layer
Invoked in Section 5.1 and Appendix B.1 as the lens for reading C as the input factor. This is a standard approximation, not an exact identity for transformer MLPs.
ad hoc to paper The model's self-generated distribution with a single random seed approximates the training/operative distribution (Eq. 4)
Appendix A.1 Eq. 4: E_{x~pθ(·|s)}[φφ^T] ≈ E_{x~Dtrain}[φφ^T]. This is the central assumption of MOIR; its validity is only empirically asserted, and the paper's Limitations admit failure modes under extreme RLHF.
standard math Activations are bounded: Mℓ := sup ∥φℓ(x)∥₂ < ∞
Appendix B.4; holds via layer normalization in modern transformers. Used for Lemma 3.
domain assumption The post-training (SFT/DPO) distribution is the correct target to preserve for deployed models
Section 5.2 and Figure 3 argue that the operative manifold shifts after post-training, but no formal justification is given for why preserving this shifted distribution is optimal; it is a design goal.

pith-pipeline@v1.3.0-alltime-deepseek · 27525 in / 14469 out tokens · 136219 ms · 2026-08-02T14:25:18.976575+00:00 · methodology

0 comments

read the original abstract

While language models remain frozen at their training state, the world evolves continuously. Knowledge editing has emerged as a key alternative to full retraining, but its deployment is bottlenecked by the erosion of core capabilities: mathematical and programmatic reasoning collapse while encyclopedic recall remains intact. We trace this asymmetric degradation to a distributional mismatch. Covariance-based editors preserve only the subspaces spanned by their reference corpus, but fail to capture the operative distribution shaped by post-training such as SFT and DPO. Static external corpora, including Wikipedia and even the original pretraining mixture, cannot recover this shifted manifold. We propose Moir, which estimates the preservation covariance $C$ directly from the model itself by sampling from its own decoding distribution. Seeding generation with a single random vocabulary token bypasses the instruction-following templates that otherwise dominate sampled outputs, exposing the broader subspaces the model has internalized. Moir requires no external data and serves as a drop-in component for any covariance-based editor, a practical advantage given that the pre- and post-training corpora of most modern LLMs are not publicly accessible. Across OLMo-2, Llama-3.1, and Qwen-3 (7-8B), under both MEMIT and AlphaEdit and in batch and sequential regimes, Moir consistently extends preservation in the most vulnerable domains, most strikingly on Qwen3-8B after 20,000 AlphaEdit batch edits, it retains 79.9% GSM8K accuracy compared to 10.9% with the Wikipedia baseline. These results suggest that aligning the preservation distribution with the model's operative distribution is a key factor in non-destructive editing, and that the model itself may be the most accessible source of that distribution for deployed systems.

Figures

Figures reproduced from arXiv: 2607.20433 by Dong-Kyum Kim, Jea Kwon, Jiwon Kim, Meeyoung Cha.

**Figure 1.** Figure 1: MOIR’s self-generation pipeline. The model autoregressively generates samples from a seed token, from which C is estimated. Using begin-of-sentence token (top) collapses generations skewed toward an instruction-style, whereas a random token (bottom) diversifies generations across domains (e.g., general text, code, and math) better approximating the model’s internalized distribution. In this work, we demons… view at source ↗

**Figure 2.** Figure 2: Differential degradation rates across knowl [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Tracking the Post-Training Manifold. Per-document NLL distributions demonstrating the clear leftward shift from the pretraining to the post-training regime. MOIR (⟨rand⟩ × 1, red) recovers this shifted operative distribution, visibly achieving a tighter match than the ⟨bos⟩-only variant (orange). Distributional Analysis. To characterize the text generated by our framework, we analyze its per-document N… view at source ↗

**Figure 4.** Figure 4: CMOIR Improves Preservation. Harmonic means over edit metrics (top) and preservation tasks (bottom), under MEMIT batch (left) and sequential (right) editing. Shades in bottom row shows the improved preservation capacity. We evaluate MOIR as a drop-in replacement for C in two representative editors: MEMIT [Meng et al., 2023] (soft preservation penalty) and AlphaEdit [Fang et al., 2025] (hard null-space p… view at source ↗

**Figure 5.** Figure 5: Per-task preservation under MEMIT. Six benchmark scores after batch and sequential [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

**Figure 6.** Figure 6: Representative radar chart on Qwen3-8B (analogous patterns for OLMo-2 and Llama-3.1 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

**Figure 7.** Figure 7: Knowledge Coverage: CMOIR vs CWiki. NLL and JSD comparisons ( [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗

**Figure 8.** Figure 8: Layer-wise indirect effect at the last subject token for all six model configurations (100 [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗

**Figure 9.** Figure 9: Base vs. instruct comparison for each architecture. Instruction tuning preserves the general [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗

**Figure 10.** Figure 10: Breaking Mode Collapse with Random-Prefix Seeding. Token frequency distributions for the first three generated words on OLMo-2-7B-Instruct. (Top) Under BOS-only seeding, generation suffers from severe mode collapse, with over 52% of trajectories trapped by the <|assistant|> chat-template attractor. (Bottom) Prepending a single uniformly-random token (⟨rand⟩ × 1) successfully breaks this post-training at… view at source ↗

**Figure 11.** Figure 11: Distributional alignment of MOIR. (a) Per-document Negative Log-Likelihood (NLL) distributions for various corpora under OLMo-2-7B-Instruct. (b) Jensen-Shannon Distance (JSD) between candidate corpora and the reference distributions. MOIR uniquely matches the post-training manifold, whereas static proxies like Wikipedia remain severely misaligned (3–4× further away). NLL Computation. For each corpus, we c… view at source ↗

**Figure 12.** Figure 12: AlphaEdit: aggregate Edit HM and Pres. HM. Pres. HM is flat for OLMo2 and Llama3, but Qwen3 with WikiText CWiki holds at ∼0.20 vs ∼0.60 with CMOIR—a gap that persists across all edit budgets and exposes AlphaEdit’s covariance dependence. 5K 10K 20K 0.0 0.5 1.0 5K 10K 20K 5K 10K 20K 5K 10K 20K 5K 10K 20K 5K 10K 20K 100 1K 0.0 0.5 1.0 100 1K 100 1K 100 1K 100 1K 100 1K CommonSense Reasoning Academic Science… view at source ↗

**Figure 13.** Figure 13: Per-task preservation under AlphaEdit. AlphaEdit is stable on OLMo2 and Llama3 regardless of C0, but on Qwen3, CWiki collapses GSM8K and HumanEval to near zero while CMOIR holds them at near-baseline levels. Batch N=5K−20K and sequential N=100−5K. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_13.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 13 linked inside Pith

[1]

Algorithm 1MOIR: Self-Generated Covariance Estimation Require:Modelp θ, target layersL, token budgetN, max sequence lengthS, vocabularyV Ensure:Sample covariances{ ˆC ℓ MOIR }ℓ∈L 1: ˆC ℓ ←0 d×d for eachℓ∈ L;n←0 2:whilen < Ndo 3:Sample seed tokenz∼Uniform(V){rand×1} 4:Generatex 1:T ∼p θ(· |z)withT≤S{Autoregressive sampling} 5:foreachℓ∈ Ldo 6:Extract keysk ...

2025
[3]

mostly preserved

HellaSwag and ARC-Challenge use length-normalized accuracy ( acc_norm) to correct for vary- ing completion lengths among multiple-choice options, following the standard protocol in lm-evaluation-harness [Gao et al., 2024]. GSM8K uses strict string matching of the final numerical answer rather than flexible extraction, which provides a more conservative es...

2024
[6]

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy

URLhttps://zenodo.org/records/12608602. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. InEmpirical Methods in Natural Language Processing (EMNLP),

arXiv
[7]

The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

Pith/arXiv arXiv
[8]

OLMo: Accelerating the science of language models

Dirk Groeneveld, Iz Beltagy, Pete Walsh, et al. OLMo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838,

Pith/arXiv arXiv
[9]

Model editing can hurt general abilities of large language models.arXiv preprint arXiv:2401.04700, 2024a

Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing can hurt general abilities of large language models.arXiv preprint arXiv:2401.04700, 2024a. Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing harms general abilities of large language models: R...

Pith/arXiv arXiv
[10]

Beyond the covariance trap: Unlocking generalization in same-subject knowledge editing for large language models.arXiv preprint arXiv:2603.15518,

Xiyu Liu, Qingyi Si, Zhengxiao Liu, Chenxu Yang, Naibin Gu, and Zheng Lin. Beyond the covariance trap: Unlocking generalization in same-subject knowledge editing for large language models.arXiv preprint arXiv:2603.15518,

arXiv
[12]

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi

arXiv:2501.00656. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: an adversarial winograd schema challenge at scale.Commun. ACM, 64(9):99–106,

Pith/arXiv arXiv
[13]

Dolma: An open corpus of three trillion tokens for language model pretraining research.arXiv preprint arXiv:2402.00159,

Luca Soldaini, Rodney Kinney, Akshita Bhagia, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research.arXiv preprint arXiv:2402.00159,

Pith/arXiv arXiv
[14]

URLhttps://github.com/togethercomputer/RedPajama-Data. Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, Kangwei Liu, Yuansheng Ni, Guozhou Zheng, and Huajun Chen. Easyedit: An easy-to-use knowledge editing framework for large language models. arXiv preprint arXiv:2308.07269,

Pith/arXiv arXiv
[17]

The single-vector case (k∗ ∈R din, b∈R dout) follows identically, yielding∆W ⋆ =b(A −1k∗)⊤/(k∗⊤A−1k∗)

The residual satisfies Tr(G δW A δW⊤) =∥G 1/2 δW A1/2∥2 F ≥0, with equality forcing δW= 0 when G≻0 . The single-vector case (k∗ ∈R din, b∈R dout) follows identically, yielding∆W ⋆ =b(A −1k∗)⊤/(k∗⊤A−1k∗). Second proof (Lagrangian; PD case directly, PSD by continuity). Assume first G≻0 . The La- grangian L= Tr(G∆W A∆W ⊤)−Tr Λ⊤(∆W Knew −R) has stationarity c...

2024
[18]

as our main method with N= 100K , matching the conventional sample size, unless otherwise noted (e.g.,⟨bos⟩). •C OLMoMix (OLMo-2 only): C computed from the actual OLMo-Mix-1124 pretraining mixture (which we refer to as the oracle) as a weighted sum of component-wise covariances (construction detailed in Appendix A.2). 19 C.2 Implementation and Reproducibi...

2020
[20]

digital natives

The integer must be ... 4 Concept <|assistant|>The term “digital natives” refers to individuals who were born during or after the digital age–-roughly those who came ... 5 Math 4c \] To calculate the product: \[ 4 \times 3^4 \] We first calculate $3^4$: \[ 3^4 = 3 \times ... 6 Code <|contador>++; That command would increment the counter (“contador”) eac...

arXiv
[2018]

Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

Pith/arXiv arXiv
[2019]

A comprehensive study of knowledge editing for large language models.arXiv preprint arXiv:2401.01286,

Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. A comprehensive study of knowledge editing for large language models.arXiv prepr...

Pith/arXiv arXiv
[2021]

Vinaik Chhetri, A. B. Siddique, and Umar Farooq. Understanding robustness of model editing in code LLMs: An empirical study.arXiv preprint arXiv:2511.03182,

Pith/arXiv arXiv
[2023]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

Pith/arXiv arXiv
[2024]

Qwen technical report.arXiv preprint arXiv:2309.16609,

Jinze Bai, Shuai Bai, Yunfei Chu, et al. Qwen technical report.arXiv preprint arXiv:2309.16609,

Pith/arXiv arXiv
[2025]

Think you have solved question answering? try ARC, the AI2 reasoning challenge

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457,

Pith/arXiv arXiv
[2026]

Perturbation-restrained sequential model editing.arXiv preprint arXiv:2405.16821,

Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hua Ling, and Jia-Chen Gu. Perturbation-restrained sequential model editing.arXiv preprint arXiv:2405.16821,

Pith/arXiv arXiv

[1] [1]

Algorithm 1MOIR: Self-Generated Covariance Estimation Require:Modelp θ, target layersL, token budgetN, max sequence lengthS, vocabularyV Ensure:Sample covariances{ ˆC ℓ MOIR }ℓ∈L 1: ˆC ℓ ←0 d×d for eachℓ∈ L;n←0 2:whilen < Ndo 3:Sample seed tokenz∼Uniform(V){rand×1} 4:Generatex 1:T ∼p θ(· |z)withT≤S{Autoregressive sampling} 5:foreachℓ∈ Ldo 6:Extract keysk ...

2025

[2] [3]

mostly preserved

HellaSwag and ARC-Challenge use length-normalized accuracy ( acc_norm) to correct for vary- ing completion lengths among multiple-choice options, following the standard protocol in lm-evaluation-harness [Gao et al., 2024]. GSM8K uses strict string matching of the final numerical answer rather than flexible extraction, which provides a more conservative es...

2024

[3] [6]

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy

URLhttps://zenodo.org/records/12608602. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. InEmpirical Methods in Natural Language Processing (EMNLP),

arXiv

[4] [7]

The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

Pith/arXiv arXiv

[5] [8]

OLMo: Accelerating the science of language models

Dirk Groeneveld, Iz Beltagy, Pete Walsh, et al. OLMo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838,

Pith/arXiv arXiv

[6] [9]

Model editing can hurt general abilities of large language models.arXiv preprint arXiv:2401.04700, 2024a

Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing can hurt general abilities of large language models.arXiv preprint arXiv:2401.04700, 2024a. Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing harms general abilities of large language models: R...

Pith/arXiv arXiv

[7] [10]

Beyond the covariance trap: Unlocking generalization in same-subject knowledge editing for large language models.arXiv preprint arXiv:2603.15518,

Xiyu Liu, Qingyi Si, Zhengxiao Liu, Chenxu Yang, Naibin Gu, and Zheng Lin. Beyond the covariance trap: Unlocking generalization in same-subject knowledge editing for large language models.arXiv preprint arXiv:2603.15518,

arXiv

[8] [12]

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi

arXiv:2501.00656. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: an adversarial winograd schema challenge at scale.Commun. ACM, 64(9):99–106,

Pith/arXiv arXiv

[9] [13]

Dolma: An open corpus of three trillion tokens for language model pretraining research.arXiv preprint arXiv:2402.00159,

Luca Soldaini, Rodney Kinney, Akshita Bhagia, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research.arXiv preprint arXiv:2402.00159,

Pith/arXiv arXiv

[10] [14]

URLhttps://github.com/togethercomputer/RedPajama-Data. Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, Kangwei Liu, Yuansheng Ni, Guozhou Zheng, and Huajun Chen. Easyedit: An easy-to-use knowledge editing framework for large language models. arXiv preprint arXiv:2308.07269,

Pith/arXiv arXiv

[11] [17]

The single-vector case (k∗ ∈R din, b∈R dout) follows identically, yielding∆W ⋆ =b(A −1k∗)⊤/(k∗⊤A−1k∗)

The residual satisfies Tr(G δW A δW⊤) =∥G 1/2 δW A1/2∥2 F ≥0, with equality forcing δW= 0 when G≻0 . The single-vector case (k∗ ∈R din, b∈R dout) follows identically, yielding∆W ⋆ =b(A −1k∗)⊤/(k∗⊤A−1k∗). Second proof (Lagrangian; PD case directly, PSD by continuity). Assume first G≻0 . The La- grangian L= Tr(G∆W A∆W ⊤)−Tr Λ⊤(∆W Knew −R) has stationarity c...

2024

[12] [18]

as our main method with N= 100K , matching the conventional sample size, unless otherwise noted (e.g.,⟨bos⟩). •C OLMoMix (OLMo-2 only): C computed from the actual OLMo-Mix-1124 pretraining mixture (which we refer to as the oracle) as a weighted sum of component-wise covariances (construction detailed in Appendix A.2). 19 C.2 Implementation and Reproducibi...

2020

[13] [20]

digital natives

The integer must be ... 4 Concept <|assistant|>The term “digital natives” refers to individuals who were born during or after the digital age–-roughly those who came ... 5 Math 4c \] To calculate the product: \[ 4 \times 3^4 \] We first calculate \(3^4\): \[ 3^4 = 3 \times ... 6 Code <|contador>++; That command would increment the counter (“contador”) eac...

arXiv

[14] [2018]

Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

Pith/arXiv arXiv

[15] [2019]

A comprehensive study of knowledge editing for large language models.arXiv preprint arXiv:2401.01286,

Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. A comprehensive study of knowledge editing for large language models.arXiv prepr...

Pith/arXiv arXiv

[16] [2021]

Vinaik Chhetri, A. B. Siddique, and Umar Farooq. Understanding robustness of model editing in code LLMs: An empirical study.arXiv preprint arXiv:2511.03182,

Pith/arXiv arXiv

[17] [2023]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

Pith/arXiv arXiv

[18] [2024]

Qwen technical report.arXiv preprint arXiv:2309.16609,

Jinze Bai, Shuai Bai, Yunfei Chu, et al. Qwen technical report.arXiv preprint arXiv:2309.16609,

Pith/arXiv arXiv

[19] [2025]

Think you have solved question answering? try ARC, the AI2 reasoning challenge

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457,

Pith/arXiv arXiv

[20] [2026]

Perturbation-restrained sequential model editing.arXiv preprint arXiv:2405.16821,

Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hua Ling, and Jia-Chen Gu. Perturbation-restrained sequential model editing.arXiv preprint arXiv:2405.16821,

Pith/arXiv arXiv