{"id":"6b300b08-06fa-4194-8aa1-f863eaa876c9","arxiv_id":"2505.19383","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"CaseEdit supplies 900 household-object commonsense edits with 3,600 multiple-choice questions and reports that AlphaEdit beats ROME, MEND, MEMIT, and MEMIT-CSK at preserving unrelated knowledge in a 3B model.","lead":"CaseEdit is a new benchmark and generation pipeline that creates personalized, atypical commonsense edits for household objects and tests whether small language models can absorb them. It matters because lightweight, edge-deployed assistants could adapt to household-specific knowledge if such edits can be made without damaging the model's other knowledge.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Base model row in Table 4 reports 0.00 locality, which is impossible under the metric's own definition; the MCQ answer-key is likely keyed to the new ground truth for all four axes, invalidating the 'minimal ripple effects' claim for AlphaEdit.","rationale":"The single load-bearing condition for the strongest claim is a trustworthy locality and portability measurement. Table 4's base-model row—0.00 on every metric—is internally inconsistent with the definition of locality in Section 4.2: an unedited model should answer unrelated questions correctly. The most economical explanation is that the evaluation harness always treats the new ground truth as the correct option, which is valid for reliability and generalization but reverses the semantics of locality and portability. If so, AlphaEdit's 'minimal ripple effects' are not measured at all, and its margin over ROME, MEND, and MEMIT could be an artifact of a scoring rule that rewards over-application of the edit. This is a correctness risk, not a disagreement with consensus; the paper itself concedes in Section 7 that no human evaluation was conducted. I agree with the reader's weakest_assumption: the paper never specifies how the correct option is assigned for locality and portability questions, and the base-model row strongly suggests the mapping is broken. The dataset-generation pipeline is transparent and the limitations section is candid, but those virtues do not rescue the headline empirical claim. The proposed concrete test—recomputing base-model locality with a correct conventional answer key—would settle the issue. If the base model then scores near 1.0, all edited-method numbers must be recomputed; if it still scores 0.0, the MCQ choices are degenerate. Either way, the current version does not support the abstract's claim, so I recommend REJECT for the current submission while leaving the door open for a corrected resubmission.","tokens_in":11772,"tokens_out":7049,"duration_ms":70135,"concrete_test":"Release the evaluation script and item-level answer keys for CaseEdit. Then take the 50 locality questions used for Table 4, run the unedited LLaMA-3.2-3B-Instruct model, and compute accuracy against the option representing the conventional unrelated answer (e.g., chef's knife → cutting food). If the base model scores near 1.0 on these items, the current 0.00 row is a scoring artifact and all reported locality and portability numbers must be recomputed; if it scores 0.0 even with a correct mapping, the MCQ choices are degenerate. Either way, Table 4 must be revised before the AlphaEdit claim can be evaluated.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central empirical claim—that AlphaEdit outperforms all baselines on CaseEdit with minimal ripple effects—rests on the four MCQ metrics in Table 4. The Base Model row reports 0.00 ± 0.00 on every metric, including Locality. This is internally inconsistent with the metric's definition in Section 4.2: locality measures whether an edit leaves unrelated inputs unchanged, so an unedited model should answer an unrelated question such as 'What is a chef's knife used for?' correctly (Table 2). A 0.00 locality score for the base model is therefore not a plausible measurement of ripple effects; it indicates the evaluation's correct-answer assignment is broken. Section 3.2 says only that the previous ground truth, the new ground truth, and three distractors are 'curated and randomly assigned labels'—it never specifies which option is correct for locality and portability questions. The most consistent explanation for a uniform 0.00 base row across all four axes is that the new ground truth is always scored as the correct answer. For reliability and generalization that is legitimate, but for locality and portability it inverts the intended semantics: a model that preserves unrelated knowledge would be penalized, and a model that over-applies the edit would be rewarded. Consequently, Table 4 and Figure 2 do not support the claim of minimal ripple effects, and the relative ranking of AlphaEdit over MEMIT, ROME, and MEND could be an artifact of the scoring rule rather than of null-space projection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CaseEdit, a dataset and generation pipeline for evaluating localized, personalized commonsense knowledge editing in small language models. The dataset is built on ATOMIC2020 and uses GPT-4o-mini to generate atypical household contexts, new ground-truth statements, and four evaluation questions per edit, covering reliability, generalization, locality, and portability in a multiple-choice format. The authors evaluate AlphaEdit, ROME, MEND, MEMIT, and MEMIT-CSK on the dataset, reporting that AlphaEdit outperforms all baselines on a LLaMA 3.2 3B-Instruct model with minimal ripple effects, and that this advantage persists in scalability tests up to 200 sequential edits.","tokens_in":12131,"tokens_out":5455,"duration_ms":48914,"significance":"If the dataset and evaluation are valid, CaseEdit addresses a real gap: existing knowledge-editing benchmarks focus on factual recall, while personalized commonsense editing in small-parameter models is underexplored. The pipeline is transparent and largely reproducible, with prompts and dataset statistics included in the appendix, and the four-axis evaluation follows standard knowledge-editing metrics. However, the central empirical claim about AlphaEdit's superiority and minimal ripple effects rests entirely on the MCQ scoring, and the reported base-model scores indicate a likely flaw in the correct-answer assignment. Because that flaw undermines Table 4 and Figure 2, the significance of the comparative results cannot be assessed until the evaluation is corrected.","major_comments":[{"comment":"The Base Model row reports 0.00 ± 0.00 on all four metrics, including Locality. Under the definition in §4.2, locality measures whether an edit leaves unrelated inputs unchanged, so an unedited model should answer a question such as 'What is a chef's knife used for?' correctly. While 0.00 for reliability, generalization, and portability might be expected for an unedited model that does not know the new ground truth, 0.00 for locality is not. A uniform 0.00 across every metric strongly suggests that the correct-answer key is always the newly generated ground truth, rather than the metric-appropriate answer. Section 3.2 specifies only that previous ground truth, new ground truth, and distractors are 'curated and randomly assigned labels from A to E,' and it never defines which option is correct for locality and portability questions. This scoring ambiguity invalidates the 'minimal ripple effects' claim and the relative ranking of methods in Table 4 and Figure 2.","section":"Table 4; §4.2"},{"comment":"The experimental setup is inconsistent about the base model: §4.3 states 'AlphaEdit, ROME, MEND, MEMIT on LLaMA 3 8B,' while the abstract, §4.1, and §5 state LLaMA 3.2 3B-Instruct. The manuscript must specify which model was actually edited; if it is LLaMA 3 8B, the abstract's claim about a 3B model is unsupported, and if it is LLaMA 3.2 3B, Section 4.3 contains a typo that must be corrected before the results can be reproduced.","section":"§4.3; §4.1"},{"comment":"MEMIT-CSK is evaluated on GPT-2 XL (1.5B parameters) while AlphaEdit, ROME, MEND, and MEMIT are evaluated on a LLaMA 3.2 3B model. Table 4 nevertheless reports all techniques in a single comparison, and §5.1 concludes that AlphaEdit 'outperforms all other knowledge editing methods.' This cross-model comparison is confounded by architecture and parameter count; the claim of outperformance over MEMIT-CSK is not valid unless all methods are run on the same base model.","section":"§4.1; Table 4"},{"comment":"The same GPT-4o-mini model generates the atypical ground truths and the evaluation questions and answer choices, and Section 7 acknowledges that no human evaluation was conducted. This creates a circularity risk: the MCQ may measure whether the edited model reproduces GPT-4o-mini's own phrasing rather than whether the edit is plausible commonsense. The paper should provide human adjudication of a sample of edits and questions, or use independently sourced answer labels, to rule out this alternative interpretation.","section":"§3.2; §7"},{"comment":"No statistical significance tests are reported, and the description of the error bars is missing: the paper does not state whether the means and standard deviations come from multiple random seeds, repeated sampling of 50 subjects, or some other procedure. For example, AlphaEdit's reliability is 0.93 ± 0.02 versus MEMIT's 0.90 ± 0.02; without knowing whether these are paired over the same 50 subjects, it is impossible to assess whether AlphaEdit's advantage is meaningful. The paper should report paired significance tests or bootstrap confidence intervals.","section":"Table 4; §5"}],"minor_comments":[{"comment":"The sentence 'AlphaEdit is less resistant to an increasing number of commonsense edits' appears to contradict the claim that AlphaEdit degrades less than other methods; 'more resistant' is likely intended.","section":"§5.1"},{"comment":"The cross-reference 'See Appendix 4' should point to the correct appendix; the parameter-size data actually appears in Appendix 8.1.","section":"§1"},{"comment":"The table caption mentions tokens highlighted in blue and red, but no highlighting is visible in the manuscript; either render the colors or remove the reference.","section":"Table 2"},{"comment":"The notation 'ATOMIC20/20' is inconsistent; the rest of the paper uses ATOMIC2020, so the abstract should be updated for consistency.","section":"Abstract; §3"},{"comment":"The fixed-edits test uses 50 subjects from CaseEdit, which contains 900 subjects; the paper should justify this sample size and state how the subjects were randomly selected.","section":"§4.3"},{"comment":"The placeholder code comments contain garbled text from PDF extraction, such as '# --- P l a c e h o l d e r v ar ia bl es ---'; the appendix should provide clean listings to be usable by readers.","section":"Appendix 8.3"}],"recommendation":"major_revision","confidential_remarks":"The base-model row in Table 4 is the most serious problem: a uniform 0.00 for locality is not plausible and points to a broken answer-key mapping. If the authors correct the scoring and rerun the experiments, the comparative conclusions could change substantially. I recommend requesting a major revision rather than rejecting, because the dataset construction and the editing comparison are both potentially valuable and the flaw is local to the evaluation design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: CaseEdit is a genuinely useful dataset idea for testing personalized commonsense editing in small models, but the evaluation is presently load-bearing broken, and the central AlphaEdit claim should not be taken at face value until the scoring rule is fixed.\n\nWhat is new: the paired typical/atypical household-object edits from ATOMIC2020 with four-axis MCQ questions is a real artifact, and the appendix shows the full generation prompts. That transparency is appreciated. The limitation section is candid about the lack of human evaluations and small scale.\n\nThe soft spots, in order of severity. First, the base model row in Table 4 reports 0.00 ± 0.00 on Locality. Under the metric's own definition in Section 4.2, an unedited model should answer an unrelated-object question correctly. A zero on Locality for the base model is impossible unless the answer key is keying every question to the new ground truth. For Locality that inverts the semantics: preserving unrelated knowledge gets penalized, over-applying the edit gets rewarded. So Table 4 and Figure 2 do not support the \"minimal ripple effects\" claim, and AlphaEdit's ranking over ROME/MEMIT/MEND may be an artifact of the scoring rule. The paper never specifies which option is correct for Locality and Portability questions (Section 3.2 just says labels are randomly assigned). That is a load-bearing gap.\n\nSecond, Section 4.3 says edits are applied on LLaMA 3 8B while everywhere else it is LLaMA 3.2 3B. And MEMIT-CSK runs on GPT-2 XL, so that comparison is confounded by both model family and scale.\n\nThird, no statistical significance tests; the standard errors are tiny but they only report ±, no test. Fourth, no public dataset or code yet, which limits reproduction.\n\nThe dataset itself is a reasonable resource, and the prompting pipeline is clearly described. But the evaluation needs a fix before the results mean anything. If the authors release the dataset and correct the scoring, this could be a useful benchmark for a real problem. As it stands, I would treat the empirical claims skeptically.\n\nI would bring it to a reading group as a cautionary tale about MCQ evaluation design, and I would consider citing the dataset once it is released and fixed. It deserves a serious referee, but the referee should focus on the scoring rule and the cross-model comparison.","headline":"CaseEdit is a genuinely useful dataset for commonsense knowledge editing, but the evaluation is currently undermined by an impossible base-model locality score that likely reflects a mis-keyed answer rule.","tokens_in":12604,"tokens_out":3855,"would_cite":false,"duration_ms":23574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CaseEdit, a new benchmark from the ATOMIC2020 graph, tests whether small models can absorb household-specific commonsense edits; the paper reports that null-space-constrained AlphaEdit does so with the least damage to unrelated knowledge.","keywords":["knowledge editing","commonsense reasoning","small language models","null-space projection","LLM personalization","ATOMIC2020","ripple effects","benchmark dataset"],"falsifier":"Take the unedited base model and run it on the locality questions from Table 4: it should answer unrelated-object questions correctly, so a 0.00 locality score for the base model would indicate the correct-answer mapping is broken rather than the edit having erased knowledge, and replacing the GPT-4o-mini-assigned answers with human-verified labels and re-scoring would settle whether AlphaEdit's margin is real.","tokens_in":11581,"feed_emoji":"🏠","tokens_out":7525,"duration_ms":63623,"temperature":0.7,"pith_summary":"CaseEdit is a dataset and generation pipeline for testing whether small language models can adopt household-specific commonsense facts, such as a butter knife being used to tighten screws in a particular home, without disturbing what they already know. The paper argues that this kind of personalized editing is exactly where small models fail, and that a benchmark built from the ATOMIC2020 commonsense graph can make the problem measurable. Using CaseEdit, the paper compares five knowledge-editing methods on a 3B-parameter model and reports that AlphaEdit, which constrains weight updates to the null space of preserved activations, outperforms the others on all four evaluation axes, including under increasing edit loads. If the results hold, lightweight models can be customized for individual households through targeted edits rather than retraining.","feed_headline":"Null-space edits beat rivals on a 3B commonsense test","feed_subtitle":"New CaseEdit benchmark shows a small model can adopt household quirks while leaving unrelated knowledge mostly intact.","key_machinery":"The object that carries the argument is the null-space projection used by AlphaEdit. When an edit is made, the method computes a weight update that satisfies the new fact while keeping the change in activation space orthogonal to the directions that encode preserved knowledge, and this is what confines the edit to the targeted association. The other load-bearing piece is the CaseEdit generation chain itself: it takes a subject from the ATOMIC2020 graph, asks GPT-4o-mini to invent an unusual but plausible household location, then asks it to generate a new use, property, or location statement conditioned on that context, and finally generates the four evaluation questions. The benchmark's statistics, including 900 subject edits and 3,600 evaluation questions, make the edits numerous enough to reveal interference between sequential updates.","core_discovery":"The central claim is that localized commonsense editing in small language models is feasible when the edit is constrained to avoid interfering with unrelated knowledge. Rather than editing arbitrary facts, the paper defines edits as paired typical and atypical uses of household objects, generated by a multi-stage prompt chain over ATOMIC2020 relation types such as ObjectUse, HasProperty, and AtLocation, with GPT-4o-mini proposing an unusual location and then a new ground-truth statement conditioned on it. Four multiple-choice questions per edit measure whether the new fact is installed (reliability), generalizes to related phrasings (generalization), leaves unrelated objects untouched (locality), and supports simple downstream reasoning (portability). On 50 sequential edits applied to LLaMA 3.2 3B-Instruct, AlphaEdit achieves the highest scores on all four metrics, with 0.93 reliability, 0.91 generalization, 0.87 locality, and 0.90 portability, and its advantage persists as the number of edits grows from 10 to 200, which the paper reads as evidence that null-space projection reduces ripple effects.","pith_inferences":["Beyond the paper's experiments, the same pipeline could generate edits for other personalization domains, such as medical routines, workplace tools, or cultural practices, by swapping the unusual-location generator for another context generator; this is an extension the paper does not test.","The reported 0.00 base-model locality score in Table 4 is a warning sign: an unedited model should answer unrelated-object questions correctly, so either the correct-answer mapping in the multiple-choice set is misaligned or the questions are not truly unaffected by edits, and a human-checked subset would resolve which.","If null-space localization transfers to other architectures, AlphaEdit-style constraints could be combined with distributed commonsense editors to get both locality and broader generalization, a hybrid direction the paper itself mentions as open."],"forward_implications":["A 3B-parameter model can internalize personalized commonsense facts with a reliability of about 0.93 after 50 sequential edits, suggesting that household customization does not require retraining or a larger model.","AlphaEdit's locality score of 0.87 means roughly nine in ten unrelated-object questions still receive the correct answer after an edit, so the ripple effect is small but not zero.","Because AlphaEdit's advantage over other methods grows as the edit count increases, null-space constraints appear to matter more as more personalized facts accumulate in one model.","CaseEdit itself provides 900 subject edits and 3,600 questions, enough to serve as a reusable benchmark for future commonsense-editing methods."],"supporting_citations":[{"why":"Introduces AlphaEdit and the null-space-constrained optimization that the paper finds best on CaseEdit.","marker":"[14]"},{"why":"Supplies the ATOMIC2020 graph structure and relation types that CaseEdit builds its editing chains on.","marker":"[16]"},{"why":"Defines the ATOMIC2020 commonsense graph and its tuple structure, the source of CaseEdit's subjects and relation templates.","marker":"[15]"},{"why":"Defines the four evaluation metrics that CaseEdit's multiple-choice questions are built around.","marker":"[22]"},{"why":"MEMIT is the scalable batch-editing baseline compared against in both the fixed and scalability tests.","marker":"[10]"},{"why":"ROME provides the rank-one editing baseline and the layer-targeting approach adapted by other methods.","marker":"[9]"},{"why":"MEND is the hypernetwork-based editing baseline evaluated on CaseEdit.","marker":"[13]"},{"why":"MEMIT-CSK is the commonsense-specific editing method that AlphaEdit must match or beat.","marker":"[12]"},{"why":"Supplies the standard implementation and default hyperparameters used to run the editing methods consistently.","marker":"[11]"}],"fun_headline_variants":["CaseEdit benchmark: AlphaEdit tops 3B commonsense","Null-space edits win on small-model commonsense test","AlphaEdit beats rivals in localized commonsense edits","New benchmark: small LLMs can learn household quirks","Small model commonsense editing: null-space wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o-mini-generated new ground truths are correct and that each multiple-choice question has a well-defined right answer; the paper's own limitations section notes that no human evaluation was conducted, so if the generated answers are implausible or the option labels are misassigned, the reported scores would not measure what they claim.","fun_headline_variants_meta":{"raw":{"variants":["CaseEdit benchmark: AlphaEdit tops 3B commonsense","Null-space edits win on small-model commonsense test","AlphaEdit beats rivals in localized commonsense edits","New benchmark: small LLMs can learn household quirks","Small model commonsense editing: null-space wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1267,"prompt_tokens":972,"completion_tokens":295,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":218}},"tokens_in":588,"tokens_out":295,"duration_ms":3500,"temperature":1.0,"reasoning_tokens":218,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:16:19.546973+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the unedited base model and run it on the locality questions from Table 4: it should answer unrelated-object questions correctly, so a 0.00 locality score for the base model would indicate the correct-answer mapping is broken rather than the edit having erased knowledge, and replacing the GPT-4o-mini-assigned answers with human-verified labels and re-scoring would settle whether AlphaEdit's margin is real.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ATOMIC2020 graph structure and relation types that CaseEdit builds its editing chains on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ATOMIC2020 commonsense graph and its tuple structure, the source of CaseEdit's subjects and relation templates."},{"cited_title":"Gupta, D","cited_arxiv_id":null,"evidence_quote":"MEMIT-CSK is the commonsense-specific editing method that AlphaEdit must match or beat."}],"review_version":1}