{"id":"4b98f3eb-33ff-4f1f-a526-46bde5a038ab","arxiv_id":"2506.00817","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"One fine-tuned small model plus an ensemble step can update a fact across multiple large language models with a single edit, outperforming separate per-model editing.","lead":"This paper presents OnceEdit, a method that updates the same new fact across several large language models by fine-tuning one small helper model and blending its output with each big model's output. It reports higher accuracy and lower editing cost than per-model editing methods, which matters for keeping AI systems current without retraining them.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline reproduction is the load-bearing weakness: MEND scores 0.00 everywhere and ROME is near 0.00 in Table 1, so the claimed consistent outperformance is not yet supported.","rationale":"The paper proposes a plausible ensemble-based mechanism for multi-model knowledge editing, and the dynamic-weight idea is a reasonable extension of DEEPEN. However, the empirical claim that ONCEEDIT 'consistently outperforms existing methods' depends on a fair set of baselines. The near-zero MEND and ROME results in Table 1 are a serious red flag: such values are not credible for these well-established methods and suggest that the baseline hyperparameters, implementations, or evaluation setup are broken. If the baselines are not correctly reproduced, the reported superiority margins are inflated regardless of how well the dynamic-weight token generalizes. The reader's identified weakest assumption about alpha transferring across target models is legitimate, but it is not the most load-bearing weakness; the primary issue is that the central comparative claim cannot be validated until the baseline reproduction problem is resolved. The paper also selectively reports validation-generation results, which further undermines the 'consistently' wording. The method itself may still be useful, so I agree with a conditional verdict: acceptance should require code release, reproducible baseline runs, and full reporting across all baselines in both evaluation settings.","tokens_in":16359,"tokens_out":7903,"duration_ms":78842,"concrete_test":"Run MEND and ROME on the same 1,000-instance ZsRE/Counterfact subsets using the official EasyEdit repository with the default configs for GPT-J-6B, Llama2-7B, and Mistral-7B-v0.1, then recompute Table 1 with the corrected numbers. If MEND/ROME recover to published levels, the claimed margins shrink; if they remain ~0, the paper must explain the failure rather than reporting it as a baseline.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim, 'consistently outperforms existing methods,' rests on Table 1 and Appendix Table 7. In Table 1, MEND is reported as exactly 0.00 on every metric for every model on both ZsRE and Counterfact, and ROME is near 0.00 (e.g., 0.07/0.06/0.01 on Llama2-7B ZsRE). These values are far below standard EasyEdit results and indicate failed or misconfigured baseline runs, not meaningful comparisons. Because MEND and ROME are dragged to zero, the averaged Score and the claimed superiority margins are inflated. The validation-generation comparison (Figure 3, Table 7) is also selective: it reports only MEMIT, WISE, GRACE, and ONCEEDIT, omitting FT-L, MEND, ROME, and DEFER, so the claim of consistent outperformance in that setting is based on a subset. The dynamic-weight classifier concern is real and worth testing, but it is secondary: even with a perfectly reliable alpha, the headline empirical claim is not established until the baselines are reproduced from their official code under standard hyperparameters.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OnceEdit, a plug-in-model-based ensemble method for updating several LLMs with a single edit. A TinyLlama plug-in is fine-tuned on edit data with an auxiliary [WEIGHT] token trained by binary cross-entropy to predict edit relevance; at inference, its sigmoid output is used as the ensemble weight α in a DEEPEN-style heterogeneous ensemble, together with zero-vector initialization and one-hot target augmentation for the decoding search. The authors report teacher-forced results on ZsRE and Counterfact for Llama2-7B, Mistral-7B-v0.1, and GPT-J-6B, validation-generation results on a subset of baselines, ablations, editing-time comparisons, FLOPS estimates, and extensions to four additional models.","tokens_in":16577,"tokens_out":5137,"duration_ms":47745,"significance":"If the empirical claims are reproducible, the work offers a useful efficiency result: one edited plug-in model can transfer a knowledge edit to multiple heterogeneous LLMs, amortizing per-model editing costs. The explicit FLOPS derivation in Appendix C and the extension to Llama3-70B are constructive. However, the current evidence does not yet support the headline claim, because two central baselines (MEND and ROME) are reported at or near zero in Table 1, which is inconsistent with published results for these methods, and the validation-generation comparison omits several baselines. The conceptual contribution of dynamic ensemble weighting via an edited plug-in is interesting and testable, but the evaluation needs to be rerun or substantially clarified before the comparative claims can be accepted.","major_comments":[{"comment":"Table 1 reports MEND as 0.00 on every metric for every model and dataset, and ROME as at most 0.07; these values are far below published results for these methods on the same benchmarks and are characteristic of failed or misconfigured baseline runs rather than meaningful comparisons. Because the claimed margins over the second-best method (14% on ZsRE, 6% on Counterfact) and the conclusion that OnceEdit 'consistently outperforms' are computed against these numbers, the central empirical claim is not currently supported. Please rerun MEND and ROME from their official implementations under EasyEdit default hyperparameters, report the exact configuration and any convergence diagnostics, and either restore plausible baseline values or explain why these runs are valid.","section":"Table 1, §4.2"},{"comment":"The validation-generation evaluation used to claim consistent outperformance reports only MEMIT, GRACE, WISE, and OnceEdit, and Table 7 shows only Llama2-7B and Mistral-7B-v0.1, omitting FT-L, MEND, ROME, and DEFER that appear in the teacher-forced comparison. The sentence in §4.2 that OnceEdit 'exhibits superior editing capabilities, outperforming all other methods' therefore cannot be evaluated for the omitted baselines; either provide the full comparison for all baselines and all three target models or explicitly restrict the claim to the evaluated subset.","section":"Figure 3, Table 7, §4.2"},{"comment":"The [WEIGHT] token is trained only on TinyLlama logits, yet at inference its sigmoid output α is applied to every target LLM in Eq. (8); the paper provides no analysis of α's accuracy or calibration on the target models. Since a mispredicted α either suppresses the edit (if too low on edit-related input) or overrides the LLM with plug-in knowledge (if too high on unrelated input), the reliability and locality claims depend on cross-model generalization that is never directly measured. Please report, for each target model and dataset, the distribution or accuracy of predicted α on edit-related and non-edit-related inputs, or otherwise validate the proposed dynamic weighting empirically.","section":"§3.1–§3.2, Eq. (6) and Eq. (8)"},{"comment":"The ablation in Table 2 shows DEEPEN with locality values of 0.02–0.14 on all models, while adding the dynamic weight raises locality to 0.96–0.99 on several rows; this pattern suggests that the DEEPEN baseline may be misconfigured for the knowledge-editing setting, and the ablation does not isolate whether the gain comes from the proposed mechanisms or from fixing an inappropriate decoding initialization. Please report the DEEPEN configuration used, including the search initialization and the number of gradient steps in Eq. (9), and confirm that the base model's outputs are preserved when no edit is applied.","section":"§5.1, Table 2"}],"minor_comments":[{"comment":"Equation (6) has an unmatched parenthesis in 'BCE (z, ϕ(logitw(x))'; the closing parenthesis for the BCE term is missing.","section":"Eq. (6)"},{"comment":"The method name is typeset inconsistently as ONCEEDIT, ONCE EDIT, and OnceEdit; please standardize to a single form.","section":"Throughout"},{"comment":"The reference entry for 'Meta 2024' includes the editorial note 'There is no corresponding record for this reference'; this entry should be completed or removed.","section":"References"},{"comment":"The FLOPS estimate assumes Csqrt = 2 and Cexp = 25 without sensitivity analysis; since these constants enter the final total, the authors should either state that the comparison is only order-of-magnitude or vary the constants to show robustness.","section":"Appendix C"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported, and several margins are only a few points (for example, ZsRE GPT-J-6B Avg. 0.87 vs. 0.89 for MEMIT in Table 1); a single run is insufficient to support the claimed stability and consistent improvement.","section":"All experimental tables"},{"comment":"Equations (12) and (13) use exact-match accuracy for generated outputs; because surface-form variation can affect exact match, the authors should specify whether normalization (e.g., lowercasing or punctuation stripping) is applied before scoring.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The core idea is coherent and the extension of DEEPEN to knowledge editing is worth reviewing, but the baseline numbers in Table 1 look like a configuration error rather than genuine results. I recommend major revision rather than rejection because the issue is empirically fixable within the scope of the paper, provided the authors rerun the baselines and report the full validation-generation comparison. I would also ask the editor to check whether the dependence on the authors' own DEEPEN work is sufficiently disclosed; the paper does cite it, but the novelty relative to DEEPEN should be stated more crisply."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely new: instead of editing each LLM separately, OnceEdit fine-tunes a small plug-in model and uses a heterogeneous ensemble to transfer the edit to several large models at once. That is a practical capability the field hasn't really had, and the paper deserves credit for formulating it cleanly. The two technical additions — the learned [WEIGHT] token for instance-level ensemble weights and the zero-initialized decoding with target augmentation — are well-motivated and the ablations show they each contribute. The method is also sensible: a small plug-in keeps the per-edit cost low, and the ensemble transfer avoids touching the big models' parameters.\n\nThe problem is the comparison. MEND scores exactly 0.00 on every metric for every model and dataset, and ROME is near 0.00 as well. Those numbers are far below anything in the published literature and strongly suggest the baselines were run with misconfigured hyperparameters or bad checkpoints, not that they genuinely failed. Since the central claim is \"consistently outperforms,\" those zeros drag down the average and inflate the winning margins. The paper also reports no error bars or multiple seeds, so we don't know if the remaining gaps are even stable. The validation-generation comparison adds to the concern: Figure 3 and Table 7 omit FT-L, MEND, ROME, and DEFER, so the \"consistent outperformance\" in that setting is based on a subset. The dynamic-weight classifier is a real worry too — alpha is trained on TinyLlama logits but used to gate all target models, and the paper never analyzes whether it generalizes. That's secondary to the baseline issue, but it still needs to be checked.\n\nWhat's solid: the method description is clear, the FLOPS derivation for the transfer matrix looks fine, and the ablation (DEEPEN -> +DW -> +DW+EE) shows each component helps. The citation to DEEPEN is self-citation but appropriate here since the ensemble backbone is their own prior work; that's not a flaw by itself.\n\nBottom line: this is a plausible, useful new application with a load-bearing empirical weakness. It deserves a serious referee, not a desk reject, but the referee should require the authors to re-run the baselines from official code with standard settings, add error bars and seeds, and analyze the learned alpha on target models. With code release and those fixes, the claim might hold; without them, the current table doesn't support it. I'd send to peer review, and yes, I'd bring it to reading group once the numbers are cleaned up.","headline":"The multi-model editing idea is fresh and worth a referee, but the implausible baseline numbers (MEND all zeros, ROME near zero) mean the headline claim of consistent outperformance isn't supported yet.","tokens_in":17138,"tokens_out":1636,"would_cite":false,"duration_ms":16178,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single edit to a small plug-in model, blended with each target LLM through a dynamically weighted ensemble, updates multiple models at once and outperforms per-model editing methods.","keywords":["multi-model knowledge editing","model ensemble","plug-in model","dynamic weight token","relative transfer matrix","knowledge editing efficiency","heterogeneous LLM ensemble","reliability generality locality"],"falsifier":"Take a trained OnceEdit plug-in and, for a target model it was not analyzed on (say Llama3-70B), record the predicted $\\alpha$ on the locality set (non-edit inputs). If $\\alpha$ is frequently greater than roughly 0.3 on inputs that should be untouched, the plug-in's unedited knowledge would pollute the target model's outputs, so the reported locality would collapse; this measurement is a direct test of whether the [WEIGHT] classifier generalizes across models.","tokens_in":16170,"feed_emoji":"🧩","tokens_out":6806,"duration_ms":57992,"temperature":0.7,"pith_summary":"Most knowledge-editing methods change one model's parameters at a time, so updating a fleet of LLMs means repeating the edit for each. This paper proposes OnceEdit: fine-tune a small plug-in model once with the new knowledge, then attach it to any target LLM by blending the two models' output distributions in a shared relative-representation space. A special [WEIGHT] token learned during fine-tuning decides per input how much to trust the plug-in versus the target model, and two decoding adjustments stop the larger target model from drowning out the edit. The paper reports that this single-edit procedure outperforms seven established editing methods on ZsRE and Counterfact across Llama2-7B, Mistral-7B-v0.1, and GPT-J-6B, while requiring only one editing intervention instead of three. If the claim holds, knowledge updates become a plug-and-play operation that is largely independent of which LLM is being updated.","feed_headline":"Edit one small model, update several LLMs at once","feed_subtitle":"OnceEdit blends a fine-tuned plug-in with each LLM via a learned weight token, beating per-model editing on ZsRE and Counterfact.","key_machinery":"The load-bearing object is the [WEIGHT] token: a token added to the plug-in model's vocabulary whose logit is sigmoid-transformed into the ensemble weight $\\alpha = \\phi(\\mathrm{logit}_w(x))$ used in Equation 8. Because it is trained jointly with the generation objective (Equation 7), it is supposed to output values near 1 for edit-related inputs and near 0 for unrelated inputs, acting as a per-instance router between plug-in knowledge and LLM knowledge. The second mechanism is the relative transfer matrix from DEEPEN, which maps each model's vocabulary distribution into a shared anchor-word space so heterogeneous LLMs can be ensembled. The third is the ensemble enhancement pair—search-space zero initialization (starting the decoding search from a zero vector rather than the LLM's distribution) and target augmentation (one-hotting the aggregated distribution)—which corrects the central-model bias that would otherwise suppress the plug-in's newly learned facts.","core_discovery":"OnceEdit's central claim is that knowledge editing can be treated as a multi-model transfer problem rather than a per-model parameter surgery. The authors train a lightweight plug-in model (TinyLlama) with a full fine-tuning objective for the new facts plus a binary classification objective on a special [WEIGHT] token, so that the token's logit, passed through a sigmoid, yields an instance-level ensemble weight $\\alpha$. At inference, the plug-in's output distribution and the target LLM's output distribution are each mapped through a relative transfer matrix of anchor-word similarities into a shared space and fused as $P = \\alpha \\cdot (p_s R_s) + (1-\\alpha) \\cdot (p_l R_l)$. Because naive ensembling lets the larger LLM dominate, the decoding search is initialized from a zero vector instead of the LLM's own distribution and the fused distribution is one-hot hardened as the target. The paper reports that this beats seven baselines on reliability, generality, and locality across three heterogeneous LLMs, and extends stably to four additional models including Llama3-70B.","pith_inferences":["If the [WEIGHT] token's alpha generalizes beyond the tested models, the same edited plug-in could act as a model-agnostic knowledge patch that is attached to any future LLM without touching its weights; this deployment pattern is an extension the paper only hints at, since it evaluates a limited set of models.","The reliance on shared anchor words between vocabularies means transfer quality is likely to depend on tokenizer overlap; a testable prediction is that performance falls smoothly as anchor-word count decreases.","One-hot target augmentation assumes the fused distribution is unimodal; edits with multiple acceptable answers might lose locality or generality, an edge case the paper does not test.","The alpha predicted by [WEIGHT] is never inspected on target models, so a direct calibration check—comparing alpha to an oracle edit-label score on each LLM—would quantify how much of the method's gain comes from the learned routing versus the ensemble enhancement tricks."],"forward_implications":["Editing TinyLlama once transfers a fact to Llama2-7B, Mistral-7B-v0.1, and GPT-J-6B, cutting the number of editing interventions from three to one and total editing time below all baselines.","On ZsRE and Counterfact, OnceEdit reports the highest average reliability–generality–locality score, with a 14% lead over the second-best method on ZsRE and 6% on Counterfact.","The result holds under both teacher-forced and validation-generation decoding, meaning the updated knowledge survives actual autoregressive generation rather than just next-token conditioning.","The method extends to four further models (Llama3-8B, Mistral-7B-v0.3, Qwen2.5-7B, Llama3-70B), suggesting the same edited plug-in transfers to newer and larger LLMs without re-editing.","The plug-in model is not unique: swapping TinyLlama for Qwen2.5-1.5B still beats all baselines, indicating the approach depends on the ensemble and weight mechanism more than on the specific small model."],"supporting_citations":[{"why":"DEEPEN provides the heterogeneous ensemble base: anchor-word relative transfer matrices that map each model's output distribution into a shared space.","marker":"(Huang et al., 2024b)"},{"why":"TinyLlama is the default plug-in model that is fine-tuned once and then ensembled with each target LLM.","marker":"(Zhang et al., 2024b)"},{"why":"EasyEdit supplies the evaluation harness and the reliability, generality, and locality metrics used in all main experiments.","marker":"(Wang et al., 2023b)"},{"why":"ZsRE is one of the two editing benchmarks; the ZsRE split adopted follows Zhang et al. (2024a).","marker":"(Levy et al., 2017)"},{"why":"Counterfact is the second benchmark, and the same paper provides the FT-L and ROME baselines and the locate-then-edit methodology.","marker":"(Meng et al., 2022a)"},{"why":"WISE is the strongest memory-based baseline and, like OnceEdit, uses unrelated knowledge as auxiliary training data.","marker":"(Wang et al., 2024b)"},{"why":"Provides the GRACE and DEFER baselines, which are compared under the validation-generation setting.","marker":"(Hartvigsen et al., 2024)"}],"fun_headline_variants":["Plug-in model edits multiple LLMs at once","One plug-in, many LLMs: ensemble knowledge editing","Multi-model editing with a single trained plug-in","Train once, edit all: plug-in for LLM updates","OnceEdit: one module to update every LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The [WEIGHT] token is trained only on TinyLlama's own inputs and logits, yet the method assumes the weight it predicts will stay trustworthy when the same plug-in is attached to a different, larger LLM at inference.","fun_headline_variants_meta":{"raw":{"variants":["Plug-in model edits multiple LLMs at once","One plug-in, many LLMs: ensemble knowledge editing","Multi-model editing with a single trained plug-in","Train once, edit all: plug-in for LLM updates","OnceEdit: one module to update every LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001181,"raw_usage":{"total_tokens":4890,"prompt_tokens":971,"completion_tokens":3919,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":3842}},"tokens_in":587,"tokens_out":3919,"duration_ms":28624,"temperature":1.0,"reasoning_tokens":3842,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:57:20.801895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained OnceEdit plug-in and, for a target model it was not analyzed on (say Llama3-70B), record the predicted $\\alpha$ on the locality set (non-edit inputs). If $\\alpha$ is frequently greater than roughly 0.3 on inputs that should be untouched, the plug-in's unedited knowledge would pollute the target model's outputs, so the reported locality would collapse; this measurement is a direct test of whether the [WEIGHT] classifier generalizes across models.","supporting_citations":[],"review_version":1}