{"id":"bb9e1162-cce7-4b3c-8898-b95635f83879","arxiv_id":"2506.12384","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"R-SFT plus task-vector scaling and pruning is proposed for knowledge editing, but the claimed sequential-editing advantage is not validated by the reported experiments.","lead":"This paper combines fine-tuning with model merging to inject many new facts into a large language model while keeping its general abilities. The authors report strong results on knowledge editing benchmarks, but the experiments do not actually test sequential editing and the numbers are inconsistent across tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sequential editing claim is unsupported: R-SFT is a batch fine-tuning procedure over the full edit set, with no edit order, incremental application, or per-edit evaluation defined, so the headline comparison against ROME/MEMIT/LoRA is not a sequential-editing comparison.","rationale":"The reader's strongest claim is that the paper's headline is about sequential editing superiority. The reader's weakest assumption is that fine-tuning on the full set at once is a valid proxy for sequential editing, and that the baselines were run under the same protocol. My stress test confirms this is the most load-bearing point: Algorithm 1 and Section 2.2 describe a batch pipeline with no sequential structure, and the experiments contain no per-edit evaluation or defined edit order. This is not a matter of minor experimental detail; it determines whether the central claim in the abstract and Section 3.2 is tested at all. I also note internal inconsistencies between Table 1 and Table 4 (e.g., C-Eval for R-SFT is 31.43 in Table 1 but 41.60 in Table 4; for Merged it is 79.35 in Table 1 but 68.42 in Table 4), which further reduce confidence in the reported numbers. These issues reinforce, rather than replace, the primary concern. Because the central claim is untested by the described experiments, the rejection verdict is appropriate. No significant additional objection beyond the reader's analysis was identified, so the verdict remains unchanged.","tokens_in":12977,"tokens_out":3061,"duration_ms":35638,"concrete_test":"Request the sequential evaluation protocol used for all methods: the ordered edit sequence, how the proposed method is applied at each step, and the per-edit results for ROME, MEMIT, LoRA, and the proposed method. If no such protocol exists, rerun the comparison as an explicit batch multi-fact editing study and rephrase the claims accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract and Section 3.2 is that the proposed framework significantly outperforms existing methods in sequential editing. However, Algorithm 1 (R-SFT) takes the entire dataset D and iterates over all samples over multiple epochs, producing one fine-tuned model; Section 2.2 then merges that single model with the base model. This is a batch multi-fact editing pipeline, not a sequential editing procedure. No edit order is defined, no incremental application is described, and no per-edit success or retention measures are reported. Table 1 is titled 'performance comparison ... for sequential knowledge editing,' but every row reports metrics on the final model after processing the whole dataset. For the comparison to be meaningful, ROME, MEMIT, and LoRA would need to be run under the same protocol (e.g., applied one edit at a time in a defined order), but the paper does not specify how these baselines were applied or whether they were run sequentially. The relevant merging-based baselines discussed in Related Work (WISE, TIES, DARE) are also omitted from the experiments, further weakening the claimed advantage. The paper therefore does not establish that the method outperforms existing methods in sequential editing; at most, it demonstrates a batch fine-tune-and-merge recipe with high edit success on the training set. This is the load-bearing flaw: if the sequential claim is removed, the central contribution of the paper is no longer supported by the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage framework for knowledge editing in LLMs: first, robust supervised fine-tuning (R-SFT) that iterates over samples with early stopping and updates only the FFN of a single transformer layer; second, model merging via a scaled and pruned task vector (Eq. 6) added to the base model. Experiments on ZsRE and several KnowEdit datasets with Qwen2.5-7B-Instruct compare the method against ROME, MEMIT, LoRA, and plain SFT, reporting edit-success, generalization, portability, locality, fluency, and general-capability benchmarks (C-Eval, CoQA, DROP, SQuAD 2.0, LogiQA). The paper claims that the framework significantly outperforms existing methods in sequential editing while better preserving general model capabilities.","tokens_in":13323,"tokens_out":3973,"duration_ms":45842,"significance":"If the results held, the proposed recipe would be a simple, architecture-agnostic baseline for injecting a large set of factual edits while retaining general capabilities, with practical value. The authors ship code and build on standard toolkits (LLaMA Factory, mergeKit, EasyEdit), which aids reproducibility. However, the central sequential-editing claim is not actually tested by the paper's protocol, and there are internal inconsistencies in the reported numbers. As a result, the significance of the contribution as stated is not established; the method may still serve as a useful batch-editing baseline, but the manuscript's current claims outrun its evidence.","major_comments":[{"comment":"This is the load-bearing issue.","section":"§3.2 and Algorithm 1"},{"comment":"The reader's note flagged this, and it is indeed load-bearing for the general-capability preservation claim.","section":"Table 1 vs Table 4"},{"comment":"This issue affects the fairness of the comparison with baselines.","section":"§3.4 and Figures 2-3"},{"comment":"This is not a fatal flaw by itself, but it compounds the sequential-editing issue.","section":"§4.2 and Table 1"}],"minor_comments":[{"comment":"","section":"Section 2.2, Eq. (6)"},{"comment":"","section":"Section 4.1 and References"},{"comment":"","section":"Table 3(a)"},{"comment":"","section":"Figure 1"},{"comment":"","section":"Section 3.1.3 and Appendix A.2"}],"recommendation":"reject","confidential_remarks":"The paper's central claimed contribution—superior sequential editing—is not supported by the experimental design, and the internal numerical inconsistencies between Table 1 and Table 4 suggest that the evaluation pipeline may contain errors. In addition, the method's novelty is limited relative to existing merging-based approaches, and the most relevant baselines are omitted. These issues are substantial and would require major new experiments and a re-framing of the contribution to be addressed; the current manuscript does not meet the bar for publication in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this one. First, the headline claim — outperforming existing methods at sequential editing — is not actually tested. Algorithm 1 takes the full edit set and fine-tunes on it for multiple epochs with sample-wise early stopping; there is no edit order, no incremental application, no per-edit retention metric. Table 1 is labeled “sequential knowledge editing,” but every row reports final-model numbers after batch training. That makes the comparison to ROME, MEMIT, and LoRA meaningless unless those baselines were run under the same sequential protocol, and the paper never says they were. Second, the numbers are internally inconsistent: R-SFT gets 78.97 on C-Eval in Table 1 but 41.60 in Table 4 for the same configuration; the merged model gets 79.35 in Table 1 and 68.42 in Table 4. That makes it hard to trust any single set of results; maybe it is a formatting error, but the burden is on the authors to explain it.\n\nWhat the paper does well: it is clearly written, it ships code, and it evaluates on the KnowEdit suite plus standard general-capability benchmarks. The two-stage recipe — early-stopped SFT restricted to one FFN layer, followed by scaling and pruning of the delta — is simple and reproducible. The ablations in Table 4 are informative: early stopping matters, and pruning helps restore some general ability. The limitations section is honest about the generalization gap.\n\nThe soft spots beyond the sequential issue: the method is essentially task-vector merging (scaling plus magnitude pruning), as in TIES, DARE, and WISE, applied to a single fine-tuned model. The paper cites those but never compares against them, which is the obvious baseline set. The hyperparameters (alpha around 0.8, sparsity 0.2) are chosen by inspecting the same benchmarks reported as results, so the “optimal” configuration is fitted to the test benchmarks.\n\nWho gets value: practitioners who want a quick batch-update recipe and do not care about the sequential framing. For a research claim about sequential editing, this paper does not provide it. With a redesign of the evaluation — an actual sequence of edits with retention measures, plus WISE/TIES/DARE as baselines — the idea might be worth a look. As it stands, the load-bearing claim fails and the reported numbers are shaky.\n\nI would still send it to peer review rather than desk-reject, because the underlying idea is sensible and the issues are fixable in principle. But I would expect the reviewers to send it back for major revision, and the authors should either drop the sequential claim or evaluate it properly.","headline":"A sensible batch fine-tune-and-merge recipe is oversold as a sequential-editing method, with internally inconsistent numbers that undercut the experiments.","tokens_in":13880,"tokens_out":4374,"would_cite":false,"duration_ms":50393,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that early-stopped fine-tuning on a single transformer layer followed by scaled, pruned merging with the base model lets a 7B LLM absorb thousands of factual edits at near-99% success while keeping general capabilities…","keywords":["knowledge editing","model merging","task vector","supervised fine-tuning","sequential editing","early stopping","large language models"],"falsifier":"Run the framework in a genuinely sequential protocol: take an ordered list of a few hundred facts, apply R-SFT and merging incrementally on slices (or one fact at a time), and after each step re-evaluate edit success on every previously edited fact; if retention of early facts falls materially below the reported ~96% success, or if ordering changes final performance, the sequential-editing claim fails.","tokens_in":12783,"feed_emoji":"🧠","tokens_out":14521,"duration_ms":134071,"temperature":0.7,"pith_summary":"The paper tries to show that knowledge editing need not choose between updating facts and preserving a language model's general abilities. It proposes a two-stage recipe: first fine-tune on all target edits at once, updating only the feed-forward networks of a single transformer layer with sample-wise early stopping; then merge the fine-tuned model back toward the original base by scaling the parameter difference and pruning it to the largest changes. On Qwen2.5-7B the recipe reaches edit success above 96% on ZsRE and most other KnowEdit subsets while C-Eval accuracy stays within about a point of the base model. The paper reads this as evidence that the approach outperforms existing editing methods in sequential editing and preserves general capabilities without architectural changes.","feed_headline":"Two-stage recipe hits ~96% edit success with general skills near base","feed_subtitle":"No architectural changes, no memory banks: one fine-tuning pass, one merge, and the model keeps both new facts and old abilities.","key_machinery":"Two mechanisms carry the argument. First, R-SFT: sample-wise consecutive gradient updates with an early-stop loss threshold, applied only to the feed-forward networks of the fifth transformer layer, so each edit is memorized without broad drift. Second, the merging identity: the task vector $\\Delta\\theta=\\theta_{\\text{sft}}-\\theta_{\\text{base}}$ is scaled by $(1-\\alpha)$ and pruned to its top-$p\\%$ parameter changes, turning fine-tuning into a sparse additive correction to the base model. The scaling/pruning step is what recovers general capability after fine-tuning degrades it.","core_discovery":"The central claim is that robust supervised fine-tuning followed by model merging solves the edit-versus-preserve trade-off: the edited model is constructed as $\\theta_{\\text{edited}} = \\theta_{\\text{base}} + (1-\\alpha)\\,\\mathrm{Top}_p(\\theta_{\\text{sft}}-\\theta_{\\text{base}})$, where $\\theta_{\\text{sft}}$ comes from sample-wise early-stopped supervised fine-tuning restricted to the feed-forward networks of a single transformer layer. The paper reports edit success of 96.95% on ZsRE, with generalization of 91.58%, while C-Eval accuracy drops from 79.57 to 79.35 and CoQA F1 actually rises from 72.60 to 75.18. It argues that the early-stopping mechanism prevents overfitting, the single-layer FFN restriction targets where facts are stored, and the scaled, pruned knowledge delta suppresses noisy updates, so the merged model retains newly acquired facts and general abilities simultaneously.","pith_inferences":["The 'sequential editing' claim is not actually tested as sequential in the paper: Algorithm 1 iterates over the full dataset, and no edit order, incremental retention measure, or per-fact forgetting check is reported, so a true sequential protocol could change the conclusion.","The method is task arithmetic with pruning and rescaling applied to knowledge editing, which opens testable refinements from the merging literature, such as pruning by sign consensus or interpolating multiple edited checkpoints.","The paper's own limitation section reports weaker transfer of edited knowledge to rephrased or multi-hop queries, so the recipe is best suited to direct factual updates; a natural extension is paraphrase augmentation or continued fine-tuning on rephrased edits to lift portability.","A practical extension of the same merge-after-fine-tune recipe would be continual learning: as new edits arrive, merge against the previously edited checkpoint instead of the base, and measure whether general ability drifts over many rounds."],"forward_implications":["If the central claim is right, any dense transformer can be updated with a single extra fine-tuning-and-merge pass, since the recipe makes no architectural changes.","Batch edits scale to thousands of facts in one run: the paper reports edit success near 99% on ZsRE with C-Eval within about a point of the base model.","The merging hyperparameters give a practical dial: scaling near 0.8 and pruning sparsity near 0.2 balance edit fidelity against general capability, per the paper's parameter analysis.","The comparison suggests that locate-then-edit methods (ROME, MEMIT) and LoRA lose most edit success when many facts are injected at once, while the proposed recipe does not."],"supporting_citations":[{"why":"Supplies the ROME baseline and the insight that factual associations concentrate in feed-forward layers, motivating single-layer FFN fine-tuning.","marker":"Meng et al. 2022"},{"why":"Supplies the MEMIT baseline for mass editing that the paper claims to outperform in sequential editing.","marker":"Meng et al. 2023"},{"why":"Supplies the LoRA fine-tuning baseline compared in the experiments.","marker":"Hu et al. 2021"},{"why":"Provides task arithmetic, the scaling-of-task-vectors operation underlying the merging formula.","marker":"Gur et al. 2023"},{"why":"Provides TIES sparsification, motivating the pruning of the knowledge delta.","marker":"Jiang et al. 2023"},{"why":"Provides DARE's parameter rescaling and pruning, cited as a related merging strategy.","marker":"Chen et al. 2023"},{"why":"Provides WISE, a prior sparsification-based knowledge editing method the paper positions against.","marker":"Wang et al. 2024b"},{"why":"Provides the KnowEdit datasets and evaluation metrics used in all experiments.","marker":"Zhang et al. 2024c"}],"fun_headline_variants":["Merge after fine-tune: edits stick, skills stay","Two-step edit: fine-tune then merge, no arch changes","Knowledge edits via model merging: 97% success, skills intact","Sequential edits solved by R-SFT plus merge","Edit and preserve: one merge keeps facts and abilities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the method excels at sequential editing rests on treating a single fine-tuning pass over the entire edit set as equivalent to editing facts one after another; the paper never defines an edit order or measures retention across a sequence, so if batch training is not a faithful proxy, the sequential comparison is untested.","fun_headline_variants_meta":{"raw":{"variants":["Merge after fine-tune: edits stick, skills stay","Two-step edit: fine-tune then merge, no arch changes","Knowledge edits via model merging: 97% success, skills intact","Sequential edits solved by R-SFT plus merge","Edit and preserve: one merge keeps facts and abilities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1415,"prompt_tokens":889,"completion_tokens":526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":443}},"tokens_in":505,"tokens_out":526,"duration_ms":5742,"temperature":1.0,"reasoning_tokens":443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:51:28.708003+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework in a genuinely sequential protocol: take an ordered list of a few hundred facts, apply R-SFT and merging incrementally on slices (or one fact at a time), and after each step re-evaluate edit success on every previously edited fact; if retention of early facts falls materially below the reported ~96% success, or if ordering changes final performance, the sequential-editing claim fails.","supporting_citations":[],"review_version":1}