{"id":"ff87af61-3637-43b2-aad6-61567426f24b","arxiv_id":"2504.20972","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SetKE couples bipartite matching with memory editing to update sets of overlapping knowledge triples in large language models, outperforming prior editors on the new EditSet benchmark.","lead":"Large language models often hold multiple true facts about the same subject, such as the continents the Roman Empire spanned, and standard knowledge editing can overwrite one fact while fixing another. This paper introduces SetKE, a method that edits sets of overlapping facts at once and reports better results than existing editors on a new benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparison is not yet controlled: baseline adaptation to set targets and set-level scoring are unspecified, so SetKE's claimed superiority may be an evaluation artifact.","rationale":"The paper's central claim is empirical: SetKE outperforms existing editors on KEO. For that claim to hold, the Table 3 comparison must measure set-editing ability rather than the ability to exploit a favorable target representation. That condition is the least secure part of the paper. The reader already identified this: Appendix A.5 defines only per-object ES/GS/LS and says the arithmetic mean is used for sets, without specifying how an object list is paired with an old set or whether the entire set must be predicted. Likewise, Section 5.1 lists baselines with no statement of their multi-object objective, so one cannot tell whether ROME/MEMIT are editing each object independently, concatenating objects, or using some other adaptation. Section 5.5's comparison between 'object set' and 'object concatenation' is billed as an ablation of bipartite matching, but it changes the target format as well, so it does not resolve the question. A re-run with a strict set-level metric and explicit baseline adaptations would settle the issue. I do not see a circularity or internal mathematical contradiction; the method is a plausible locate-then-edit extension. The verdict should remain conditional pending this control. The inconsistent Neigh.P. total in Table 2 is a secondary data-integrity note but does not change the main concern.","tokens_in":15455,"tokens_out":11910,"duration_ms":126242,"concrete_test":"Re-run Table 3 with every baseline given the same per-object target list as SetKE (e.g., one MEMIT/ROME update per target object, same layers and learning rate) and evaluate with both the paper's per-object average and a strict set-level score that requires every target object to rank above every non-target object on the edit prefix. Also recompute the Neigh.P. totals in Table 2 and clarify which prompts contribute to Locality. If SetKE still leads under the strict set-level score with corrected totals, the concern is resolved; otherwise the empirical advantage is confounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the Table 3 comparison on EditSet, but Section 5.1 never states how single-object editors (ROME, MEMIT, MEND, PMET, KN, FT-W) are adapted to multi-object targets. Appendix A.5 defines ES/GS/LS per object and then averages, without defining how a set edit like {Europe,Asia}->{Africa,Asia} is paired against the old set or whether the full target set must be generated; an editor that independently raises each target object's probability can score well even if it never produces the set as a set. Section 5.5's 'ablation' comparing 'object set' with 'object concatenation' changes both the target representation and the optimization, so it does not isolate the bipartite-matching contribution. Additionally, Table 2 reports Neigh.P. total as 3,988 while the per-overlap entries sum to 13,275, so the Locality component of the composite Score is computed on an unexplained, possibly tiny subset. Without specifying baseline adaptation and a set-level correctness criterion, the claimed superiority over existing methods may reflect the evaluation protocol rather than editing quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies Knowledge Element Overlap (KEO) in knowledge editing, focusing on cases where multiple triplets share a subject and relation (the RSO type) and therefore share a knowledge prefix. It presents a pilot analysis showing that KEO is prevalent in existing KE datasets and that current editors (ROME, MEMIT, MEND) degrade on KEO instances. It then introduces EditSet, a Wikidata-derived benchmark of KEO triplets, and proposes SetKE, a locate-then-edit method that treats the object as a set and uses bipartite matching (Hungarian algorithm) to align model predictions with the target set before applying multi-layer residual weight updates. Experiments on GPT2-Large, GPT2-XL, and GPT-J report that SetKE outperforms existing methods on EditSet under KEO scenarios, with additional results on Counterfact and ablations against object-concatenation targets.","tokens_in":15651,"tokens_out":3681,"duration_ms":39546,"significance":"If the reported comparisons are controlled, the paper addresses a real and under-studied failure mode: knowledge editing methods are typically defined for a single object, whereas many factual queries legitimately accept a set of objects. The proposed KSE formulation is a sensible generalization, EditSet could serve as a reusable benchmark, the pilot analysis is reasonable, and the authors provide code. The claim of state-of-the-art performance, however, rests on evaluation details that are not yet specified: how single-object baselines were adapted to multi-object targets, and what counts as a correct set edit. These issues are load-bearing for the central contribution, so the paper needs a revision before the results can be accepted as claimed.","major_comments":[{"comment":"Section 5.1 lists the baselines but never states how single-object editors (FT-W, KN, MEND, PMET, MEMIT, ROME) are applied to a multi-object target O={o1,...,oN}. Since these methods are defined for (s,r,o)->(s,r,o*), the reader cannot tell whether each baseline is run independently per object, with a concatenated target string, with only the first object, or with some other adaptation. The main comparison in Table 3 is therefore not yet controlled: the reported margins over MEMIT and ROME could be artifacts of baseline adaptation rather than evidence of set-level editing quality. Please specify, for each baseline, the exact edit input, target string, and loss used when N>1.","section":"Section 5.1 and Appendix A.5"},{"comment":"The Efficacy Score is defined as E_i[P[o*_i|p(s_i,r_i)] > P[o_i|p(s_i,r_i)]] and then averaged over objects, but the paper does not define set-level correctness: it is not stated whether all o*_i must receive high probability jointly, whether the old objects o_i must all be suppressed together, or whether the model must generate exactly the target set. As written, a method that independently boosts each target object over its old counterpart achieves 100% ES even if it never produces the set {o*_1,...,o*_N} as a set. This is exactly the kind of behavior that a set-editing method should be distinguished from, so a set-level success criterion or a justification for averaging per-object scores is needed.","section":"Appendix A.5"},{"comment":"Table 2 reports the Neigh.P. total as 3,988, but the per-overlap entries 3,780 + 2,768 + 2,240 + 1,757 + 1,509 + 1,221 sum to 13,275. The Locality component entering the composite Score in Tables 3 and 4 is therefore computed on an unexplained subset that appears to be roughly 3.4% of the Counter.P. prompts, while Efficacy and Generalization use 40,900 and 78,031 prompts. The composite harmonic mean is not comparable across components if LS is measured on a different and much smaller sample than ES and GS. Please correct the table or report LS on the full neighborhood-prompt set.","section":"Table 2"},{"comment":"The ablation comparing 'object set' with 'object concatenation' changes two variables simultaneously: the target representation (a set of objects vs a concatenated string) and the optimization objective (bipartite matching vs a sequence-level loss). The drop in SetKE performance from 73.68 to 59.50 in the concatenation condition could be due to the representation change alone, so the experiment does not isolate the contribution of bipartite matching. Moreover, under the concatenation setting SetKE is worse than ROME (59.50 vs 61.30), which complicates the claim that matching is the source of SetKE's advantage. Please add an ablation that fixes the target representation and varies only the matching strategy, such as SetKE with independent per-object loss versus SetKE with Hungarian matching, and report baselines under the identical set-based target encoding.","section":"Section 5.5"}],"minor_comments":[{"comment":"The paper defines four overlap types but then states that KEO 'typically refers to the RSO type' and builds EditSet entirely on RSO instances; the abstract's general KEO claim is broader than what is evaluated. Please clarify the scope of the claims.","section":"Section 2.1/2.2"},{"comment":"The text says results are averaged over 5 repetitions, but the confidence intervals in Tables 3, 4, and 7 are not described; please state whether they are bootstrap intervals over edits, standard errors over repetitions, or something else.","section":"Appendix A.2"},{"comment":"The notation yij with j=π(j) is confusing; the token index and the matching index should be separated so that L_edit^π(j) is clearly a function of the matched target and prediction.","section":"Equation (4)"},{"comment":"There are repeated typos and formatting inconsistencies, including 'with a subjects', 'the l-thlayer', and the inconsistent rendering of EditSet as 'E DIT SET'; a careful proofread would improve readability.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper identifies a real gap: RSO-type facts (one subject, one relation, multiple valid objects) are common, current knowledge-editing methods degrade on them, and the problem had not been treated as a set-editing task before. The pilot analysis in Section 3, showing the degradation on Counterfact, is the strongest part of the paper and is believable. The second thing: SetKE is a sensible engineering answer (ROME/MEMIT-style locate-and-edit with a Hungarian-matching loss borrowed from set prediction), and the method also wins on Counterfact in the appendix, which is an external benchmark. So the direction is probably right.\n\nWhat is actually new: the Knowledge Set Editing formulation, the EditSet dataset (31 relations intersecting Counterfact/Pararel, from Wikidata), and applying bipartite matching to the edit loss. The dataset is the most durable contribution; I expect people to use it.\n\nWhere it is soft, in order:\n\n1. The comparison is not controlled yet. Section 5.1 never states how ROME, MEMIT, MEND, KN, or FT-W were adapted to multi-object targets. Table 4 shows both \"object set\" and \"object concatenation\" protocols for GPT-J, but for the headline Table 3 results the protocol each baseline saw is unspecified. If a single-object editor was fed one target at a time, the gap in Table 3 is an artifact.\n\n2. The metric is per-object, not set-level. Appendix A.5 defines ES/GS/LS for one object and then averages over objects. Nothing checks whether the model produces the target set as a set; an editor that independently boosts each new object above its old counterpart scores perfectly. That is a legitimate first metric, but the paper should say so and add a set-level criterion.\n\n3. Table 2 has arithmetic problems. The Neigh.P. row sums to 13,275 but the total column says 3,988, meaning the Locality component of the composite score may be computed on an unexplained, much smaller subset. Counter.P. and Para.P. totals are also off by a few. Minor, but sloppy in a paper whose contribution is a benchmark.\n\n4. The \"ablation\" in Section 5.5 changes both the target representation and the optimization, so it does not isolate the bipartite-matching loss. And under object concatenation SetKE loses to ROME (59.50 vs 61.30 on GPT-J), which is a useful honesty point but should be discussed as such.\n\nThe stress-test concerns hold up on reading; I do not think they are manufactured. The central claim, that SetKE outperforms existing methods in KEO scenarios, is plausible but not yet demonstrated.\n\nBottom line: this is a knowledge-editing-subfield paper; the problem formulation and dataset are the parts most likely to outlive the method. It deserves a serious referee, but the referee should ask for baseline adaptation details, a set-level correctness metric, and a corrected Table 2 before the empirical claims are accepted.","headline":"The KEO problem is real and the method is sensible, but the headline comparison is not controlled: baseline adaptation and set-level scoring are underspecified, so the claimed superiority is plausible, not yet established.","tokens_in":16176,"tokens_out":6358,"would_cite":true,"duration_ms":58764,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A set-level editor fixes overlapping knowledge triplets that single-edit methods overwrite.","keywords":["knowledge editing","knowledge element overlap","knowledge set editing","bipartite matching","Hungarian algorithm","factual triplets","language model editing","EditSet benchmark"],"falsifier":"Run the EditSet comparison with a set-level success criterion that requires the post-edit model to rank every target object higher than every non-target object for the shared prefix, and that measures locality on the objects that should stay unchanged; if ROME or MEMIT with a multi-object objective then matches or beats SetKE, the paper's core claim of a formulation-level advantage is false.","tokens_in":15268,"feed_emoji":"🧠","tokens_out":7399,"duration_ms":69458,"temperature":0.7,"pith_summary":"Factual knowledge stored in a language model is usually edited one triplet at a time, but real facts often share the same subject and relation and have several valid objects (the Roman Empire spanned Europe and Asia). The paper argues that this overlap, which it calls Knowledge Element Overlap (KEO), is widespread in existing editing benchmarks and causes standard editors to overwrite each other's changes, because overlapping triplets activate overlapping neurons in the model's feed-forward layers. To fix this, the paper proposes Knowledge Set Editing (KSE): the edit target is a whole set of objects, and a new method, SetKE, aligns the model's predicted object set with the target set using bipartite matching, then updates the weights with the locate-then-edit machinery of prior methods. On a new Wikidata-derived benchmark, EditSet, the paper reports that SetKE beats prior editors on efficacy and generalization across GPT-2 Large, GPT-2 XL, and GPT-J while preserving locality. If correct, this reframes knowledge editing as a set-alignment problem rather than a single-fact rewrite.","feed_headline":"Set editing beats one-at-a-time fixes for overlapping facts","feed_subtitle":"SetKE edits all objects sharing a subject and relation at once, beating ROME and MEMIT on EditSet.","key_machinery":"The machinery is a bipartite matching constraint wrapped in the standard locate-then-edit pipeline. The model's predicted objects \\hat{y} and the editing targets y are treated as two sides of an assignment problem; the Hungarian algorithm selects the permutation \\hat{\\pi} that minimizes the sum of negative predictive probabilities -1{y_j≠∅} P_{π(j)}(y_j). That matching turns the set edit into a structured loss L = Σ 1{y_j≠∅} $L_edit^{{π(j)}}$ + L_const, where L_edit is a token-prediction loss on the matched target and L_const is a KL divergence keeping the subject's unchanged behavior intact. With the matching fixed, SetKE locates the relevant FFN layer, computes a residual vector, spreads it across several layers, and updates W_proj with the closed-form low-rank update borrowed from batch-editing methods. The matching is what distinguishes SetKE: instead of optimizing each object independently or as one long string, it first decides which current prediction should be edited into which target, preventing two targets from fighting over the same prediction.","core_discovery":"The paper's central claim is that the failure of current knowledge editors on overlapping facts is not a tuning problem but a formulation problem: knowledge editing should operate on sets of objects, not single objects. Concretely, when a subject and relation map to a set O = {o1,...,oN}, editing to O* = {o*1,...} by applying a single-object editor N times (or by concatenating targets) produces conflicting overwrites, because the shared prefix tr(s) routes all objects through the same stored association. SetKE treats the current output set and the target set as two sides of an assignment problem, finds the cheapest one-to-one matching via the Hungarian algorithm, and back-propagates an edit loss plus a locality-preserving KL constraint through the located feed-forward layers. The paper reports that this set-based loss lets SetKE outperform ROME, MEMIT, MEND, PMET, KN, and fine-tuning on the EditSet benchmark across three LLM sizes, with the largest gains on Efficacy and Generalization and roughly unchanged Locality.","pith_inferences":["A natural extension the paper does not pursue is to use the degree of neuron overlap to predict which layer and which residual should be edited, potentially removing the need to search target layers separately for each model.","The same set-alignment view could apply to any prompt with multiple valid answers, such as list-valued attributes or open-ended generation, where a single-target edit loss is ill-defined; the Hungarian-constrained loss is a candidate for those settings.","A sharper comparison would give ROME or MEMIT the same matching objective inside their own updates; the paper's concatenation ablation is a weaker adaptation, so the claimed advantage may be understated or overstated depending on how baselines are extended."],"forward_implications":["If KEO is as prevalent as the paper's dataset statistics show (as low as 4–19% of instances are non-overlapping in some benchmark datasets), then a large share of past knowledge-editing evaluations may have been measuring performance on an atypical subset.","On EditSet, SetKE improves Score over the best baseline by roughly 7–10 points on GPT-2 Large and GPT-2 XL, with Efficacy and Generalization gains up to about 39 and 35 points over MEMIT; treating the edit as a set rather than as independent triplets is the claimed reason.","All tested editors degrade as the number of overlapping objects grows from 3 to 8 or more, but SetKE degrades less than ROME and MEMIT, consistent with less knowledge overwriting under the set constraint.","The ablation shows that editing a set of objects simultaneously beats concatenating the objects into one long target, suggesting that alignment, not just parallelism, is what helps."],"supporting_citations":[{"why":"Supplies the locate-then-edit paradigm, the target-layer localization, and the editing metrics that SetKE inherits.","marker":"[Meng et al., 2022a]"},{"why":"Supplies the multi-layer residual spreading and the closed-form weight update used in SetKE's editing step.","marker":"[Meng et al., 2022b]"},{"why":"MEND is a baseline meta-learning editor whose poor set-editing performance illustrates the KEO failure.","marker":"[Mitchell et al., 2021]"},{"why":"Knowledge attribution method used to localize overlapping knowledge neurons in the overwriting analysis.","marker":"[Dai et al., 2022]"},{"why":"Hungarian algorithm used to find the optimal matching between predicted and target object sets.","marker":"[Kuhn, 1955]"},{"why":"Set prediction with bipartite matching inspires the structured loss formulation.","marker":"[Sun et al., 2021]"},{"why":"Provides the zsRE dataset, one of the datasets whose KEO prevalence is measured.","marker":"[Levy et al., 2017]"},{"why":"Provides the PARAREL dataset, used for KEO prevalence statistics and for selecting the 31 common relations in EditSet.","marker":"[Elazar et al., 2021]"}],"fun_headline_variants":["Set-wise editing fixes overlapping knowledge triplets","Edits that treat facts as sets beat single-object fixes","Overlapping facts need set-based editing, not one-by-one","SetKE edits object sets, outperforming ROME and MEMIT","For overlapping facts, edit the set, not each object"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's comparison rests on the assumption that scoring each object in a set separately and then averaging captures whether the whole set was edited correctly, and that the comparison editors were set up fairly for multi-object edits; if that scoring favors the set-based method, the main result is an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Set-wise editing fixes overlapping knowledge triplets","Edits that treat facts as sets beat single-object fixes","Overlapping facts need set-based editing, not one-by-one","SetKE edits object sets, outperforming ROME and MEMIT","For overlapping facts, edit the set, not each object"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1351,"prompt_tokens":915,"completion_tokens":436,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":355}},"tokens_in":531,"tokens_out":436,"duration_ms":4840,"temperature":1.0,"reasoning_tokens":355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:14:56.284980+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the EditSet comparison with a set-level success criterion that requires the post-edit model to rank every target object higher than every non-target object for the shared prefix, and that measures locality on the objects that should stay unchanged; if ROME or MEMIT with a multi-object objective then matches or beats SetKE, the paper's core claim of a formulation-level advantage is false.","supporting_citations":[{"cited_title":"Zero-shot relation extraction via reading compre- hension","cited_arxiv_id":null,"evidence_quote":"Provides the zsRE dataset, one of the datasets whose KEO prevalence is measured."},{"cited_title":"Calibrating factual knowledge in pretrained language models","cited_arxiv_id":null,"evidence_quote":"Knowledge attribution method used to localize overlapping knowledge neurons in the overwriting analysis."}],"review_version":1}