{"id":"d34fd84e-401b-4733-ac20-732e6b73218c","arxiv_id":"2412.11418","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An automated pipeline that verifies, conceptualizes, and edits commonsense knowledge inside LLMs reports improved plausibility and downstream QA accuracy.","lead":"ConKE is a pipeline that finds commonsense mistakes in a language model, rewrites the knowledge into broader concepts, and edits just a few internal weights to fix the errors. The authors show their edited models give more plausible commonsense answers and perform better on five reasoning benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VERA is both edit selector and evaluator, and the independent checks (200 human ratings, best-model-only downstream bars) are too weak to break the feedback loop; the plausibility gains may not generalize.","rationale":"The reader's conditional verdict is appropriate, and I agree that VERA's trustworthiness is the weakest assumption. The framework is coherent, and the conceptualization ablation in Figure 4 provides internal support for the value of abstraction. However, the central claim has two pillars with asymmetric evidence: the plausibility pillar uses VERA for both selection and evaluation, creating a feedback loop; the downstream pillar is external but reported only for the single best model with no statistical support. The 200-sample human check is too small and too qualitatively described to break the loop. My concern sharpens the specific condition that must be met: independent human or alternative-verifier validation of the plausibility gains, plus per-model downstream statistics with error bars. No fraud or misconduct is implied; the issues are evidentiary and addressable. The verdict should remain conditional on those additions.","tokens_in":11300,"tokens_out":6087,"duration_ms":56117,"concrete_test":"Draw a stratified sample of 1,000 test-set generations (250 per condition: vanilla, MEMIT, GRACE, ROME) and have at least three annotators rate plausibility with a pre-registered rubric, reporting inter-annotator agreement. Compute the edited-vs-vanilla acceptance gap under human ratings and compare it to the VERA plausible-rate gap; also compute the item-level Spearman correlation between VERA scores and human scores. If the human gap does not mirror the VERA gap (or the correlation is low), the VERA feedback loop is not validated and the plausibility claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that ConKE improves commonsense plausibility and downstream QA—requires that VERA's plausibility scores are a valid proxy for human commonsense. That condition is least secure because VERA is used twice: in §3.1 it selects which knowledge to edit, and in §4.1 it is the primary outcome metric (plausible rate >0.5). The only independent checks are (i) 200 manually reviewed generations, reported only as 'similar improvements' with no inter-annotator agreement or per-item correlation, and (ii) downstream benchmark bars in Figure 3 for a single best model with no error bars, no seeds, and no per-backbone/per-method breakdown. The ablation in Figure 4 also uses average VERA scores as the outcome. If VERA systematically prefers generic or template-like tails (e.g., 'PersonX has fun'), the conceptualization step could inflate VERA scores without improving human-judged plausibility; the 200-sample check could pass because human raters also accept generic statements, while the downstream evidence is too thinly reported to break the loop. Section 4.1's own text admits the human check is a 'subset of 200 generations,' and the Limitations section concedes there is no stable ground truth for commonsense. Without an independent, larger-scale validation, the reported plausibility gains may be optimization toward VERA rather than genuine commonsense improvement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ConKE, an automated knowledge-editing pipeline for commonsense: VERA scores are used in Section 3.1 to select implausible (head, relation, tail) generations from an LLM; GPT-4o conceptualization and instantiation in Section 3.2 expand the triples to be edited; and MEMIT, ROME, or GRACE in Section 3.3 apply the edits. Evaluation in Section 4 measures a plausible rate on AbstractATOMIC test generations using VERA, adds a 200-sample human assessment, reports downstream accuracy on five QA benchmarks for the best model, and presents an ablation comparing VERA scores with and without conceptualization. The claimed contributions are a fully automated and scalable commonsense knowledge-editing framework and improved plausibility and QA performance.","tokens_in":11480,"tokens_out":4666,"duration_ms":40634,"significance":"If the central claim holds, ConKE would be a useful contribution: it replaces human annotation for detecting implausible commonsense knowledge, uses conceptualization to increase edit generalizability, and evaluates three editors across four backbones. The release of data, code, and models is a strength, as is the use of external downstream benchmarks. However, the evidence as presented does not yet establish the central plausibility claim: the primary outcome metric, VERA, is also the model used to decide which knowledge to edit, so part of the measured gain may be optimization toward VERA's threshold. The independent human check is small, lacks agreement statistics, and is reported only descriptively, while the downstream results are given for a single unnamed best model without variance. These gaps are fixable but currently leave the headline claim under-supported.","major_comments":[{"comment":"VERA is used both to select which triples to edit (score below 0.5 in Section 3.1) and to compute the primary outcome metric (plausible ratio above 0.5 in Section 4.1), so the reported plausibility gains are partly optimization toward VERA's threshold rather than independent evidence of improved commonsense. The 200-sample human evaluation in Section 4.1 is reported only as \"similar improvements\" with no inter-annotator agreement, no per-item acceptance counts, and no correlation with VERA scores; it is too small and too thinly described to break the feedback loop. Please add a larger human evaluation with agreement statistics, or an independent plausibility metric not involved in edit selection.","section":"Section 3.1 and Section 4.1"},{"comment":"The downstream claim is based on a single \"best LLM\" that is never named, with one bar per benchmark, no error bars, no number of seeds, and no per-backbone or per-editor breakdown. Since Figure 2 shows that the editors behave differently, a single best-model comparison cannot support the abstract's claim of stronger performance across multiple QA benchmarks. Report all backbones and editors with exact accuracies and variance (or at least multiple seeds), and state whether the improvements are statistically significant.","section":"Section 4.2 and Figure 3"},{"comment":"The ablation's outcome is again average VERA scores, so it inherits the same feedback-loop concern as Sections 3.1 and 4.1. In addition, Figure 4 shows only Mistral-7B, Llama-3-8B, and ChatGLM2-6b, although Section 3.3 lists GPT-J-6B as one of the four backbones. Please include GPT-J-6B results and an evaluation metric that is independent of the edit-selection model.","section":"Section 4.3 and Figure 4"},{"comment":"The conceptualization and instantiation steps are performed by GPT-4o with no reported verification that the roughly 160,000 generated triples preserve the semantic content of the original triples or that the abstract concepts are valid generalizations. Since the method's claimed advantage rests on these augmented triples being correct and generalizable, include sample outputs and a quality check (automatic or human) of the augmented knowledge, along with the number of triples that failed generation or filtering.","section":"Section 3.2"}],"minor_comments":[{"comment":"The framework is called both \"ConceptEdit\" and \"ConKE\" in the Abstract, Figure 1, and the introduction; please use one consistent name throughout.","section":"Title and Section 1"},{"comment":"The sentence \"implausible information within the LLM. implausible information within the LLM.\" contains a duplicated fragment and should be corrected.","section":"Section 1"},{"comment":"The figure does not state how many generations were scored, how the 200-sample subset was drawn, or whether both annotators rated all items; please provide exact values and inter-annotator agreement.","section":"Figure 2"},{"comment":"The figure has no numeric values, no model name, and no indication of variance; a table with exact accuracies per benchmark would be more informative.","section":"Figure 3"},{"comment":"The text alternates between \"plausible ratio\" and \"plausible rate\"; please use one term consistently.","section":"Section 4.1"},{"comment":"The phrase \"significant performance improvements\" in Section 4.2 is not supported by any significance test or confidence interval; please soften the wording or add statistical evidence.","section":"Section 3.3 and Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the overlap between the edit-selection model and the outcome metric; I would ask the authors to address this with a substantially larger and better-reported human evaluation or an independent plausibility measure, and to provide complete per-backbone downstream results. The paper is within the journal's scope and the framework is promising, but the current evidence base is too thin for acceptance as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe paper delivers what it promises: a fully automated pipeline that diagnoses implausible commonsense beliefs in an LLM with VERA, expands them through conceptualization/instantiation using GPT-4o, and edits the model with MEMIT, ROME, or GRACE. That closed loop is genuinely new, and it is a reasonable answer to the annotation bottleneck in commonsense KE. The authors test several backbones, include an ablation, and release code and data. They also cite the relevant prior work, including the closest free-text KE method.\n\nThe main weakness is evaluator overlap. VERA chooses which knowledge to edit (§3.1) and then scores whether the edit worked (§4.1). That makes part of the reported plausibility gain an optimization toward VERA's own threshold. The downstream QA improvements are independent evidence, but they are shown only for the best LLM in Figure 3, with no error bars, seeds, or per-backbone breakdown, so they don't break the loop. The 200-generation human check is the right idea but it is too thin: two annotators, no inter-annotator agreement, only a summary sentence. The ablation in §4.3 also uses VERA as the outcome. These are not fatal flaws; the framework is coherent and the conceptualization step likely helps. But the demonstration is weaker than the abstract claims.\n\nThe Limitations section is candid about cascading edits and the absence of stable commonsense ground truth. What the paper needs is a stronger independent evaluation: a second verifier or a held-out human set for the outcome metric, plus full per-model, per-seed results on the benchmarks.\n\nI would send this to peer review, not desk reject it, but I would condition acceptance on that additional evidence. Readers working on knowledge editing or commonsense will find it a useful starting point, even if the current evidence is under-determined.","headline":"A coherent and potentially useful commonsense KE pipeline whose main plausibility result is weakened by VERA serving as both edit selector and outcome measure; independent evidence is too thin.","tokens_in":12096,"tokens_out":3679,"would_cite":true,"duration_ms":32175,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an LLM's commonsense knowledge can be repaired automatically by using a second model's plausibility scores to locate errors, abstracting the target knowledge into general concepts, and instantiating those concepts…","keywords":["knowledge editing","commonsense reasoning","conceptualization","instantiation","plausibility estimation","large language models","automated verification","question answering"],"falsifier":"Have independent raters with measured inter-annotator agreement score a sample of the triples VERA flagged as implausible and a sample of post-edit generations, then compare per-item labels with VERA's scores; if agreement is near chance, or if editing the same number of randomly chosen triples produces the same downstream gains, the reported improvements are not attributable to VERA-guided conceptualized editing.","tokens_in":10994,"feed_emoji":"🧠","tokens_out":8007,"duration_ms":65996,"temperature":0.7,"pith_summary":"ConKE's claim is that commonsense editing in LLMs can be made fully automated and scalable: instead of relying on manually curated gold labels, the pipeline uses a separate plausibility verifier (VERA) to detect which commonsense triples the target model generates implausibly, enriches those triples through conceptualization and instantiation so that each edit covers a family of related statements, and then writes the enriched knowledge back into the model with existing editing algorithms. The paper reports that edited LLMs generate more plausible commonsense according to VERA and to a 200-sample human review, and that the best edited backbone outperforms its unedited baseline on all five commonsense question-answering benchmarks tested (aNLI, SocialIQA, PIQA, CommonsenseQA, WinoGrande). A sympathetic reader would care because the approach attacks the two bottlenecks that have kept commonsense knowledge editing from scaling: there is no complete commonsense resource to copy from, and human annotation of erroneous beliefs is unaffordable at scale.","feed_headline":"Commonsense edits without labels lift five QA benchmarks","feed_subtitle":"A verifier flags implausible beliefs, conceptualization expands them, and edited models improve on five commonsense benchmarks.","key_machinery":"The load-bearing machinery is the pairing of an automated plausibility judge with conceptualization–instantiation enrichment. VERA is a discriminative model that scores the plausibility of arbitrary commonsense statements on [0,1]; ConKE uses it both to select which generated triples to edit (score below 0.5) and to measure whether editing improved plausibility. Conceptualization abstracts a concrete event such as 'PersonX plays together every day' into a general concept such as 'PersonX engages in enjoyable group activities,' and instantiation produces new concrete variants, so one edit becomes a family of edits. The editing layer is a standard knowledge editor: ROME or MEMIT modify internal weights, and GRACE adds an external key–value adapter memory.","core_discovery":"The central claim, stated on the paper's own terms, is that conceptualization-augmented editing makes a small set of detected commonsense errors generalizable into broad corrections. ConKE first prompts the target LLM to complete (head, relation) pairs from AbstractATOMIC, then uses VERA to score the generated tails and selects those scoring below 0.5 as implausible knowledge that needs editing. For each selected triple, GPT-4o abstracts the event into a general concept and instantiates it into new concrete variants, yielding roughly 160,000 enriched commonsense triples. These are applied to the model through ROME, MEMIT, or GRACE. The paper reports improved VERA plausibility on held-out AbstractATOMIC triples, a matching trend in expert acceptance on 200 sampled generations, and downstream accuracy gains over the vanilla baseline on five commonsense QA benchmarks.","pith_inferences":["Because VERA supplies both the signal that selects edits and the metric that reports plausibility gains, part of the reported improvement may be alignment with VERA rather than with human commonsense; the downstream benchmark gains are the more independent evidence, though they are reported for only the best backbone and without error bars.","The authors do not test whether the pipeline depends on the specific scorer; swapping VERA for a different plausibility model or for human-annotated implausible triples would show whether the mechanism transfers.","The paper's own limitations section notes that edits can cascade through related concepts, that repeated updates risk knowledge drift, and that commonsense lacks stable ground truth; these caveats qualify how far the results extend beyond the controlled evaluation.","Because the editing format is open-ended rather than restricted to (head, relation, tail), the same detection–enrichment–edit loop could plausibly carry to other knowledge domains, provided a suitable plausibility scorer exists."],"forward_implications":["Commonsense knowledge editing can run end-to-end without human-labeled errors, because VERA supplies the detection and evaluation signal.","One detected error becomes a family of edits, since conceptualization and instantiation expand the target triple into abstract and concrete variants, and this is what the authors claim improves generalizability.","The approach composes with existing editing algorithms and backbones: ROME, MEMIT, and GRACE all show plausibility gains across four LLMs.","If the paper is right, improving internal plausibility transfers to commonsense question answering: the best edited model outperforms the vanilla baseline on aNLI, SocialIQA, PIQA, CommonsenseQA, and WinoGrande."],"supporting_citations":[{"why":"Supplies VERA, the discriminator whose [0,1] plausibility scores select which triples to edit and evaluate the edits.","marker":"Liu et al., 2023"},{"why":"Supplies AbstractATOMIC, the abstract commonsense knowledge base whose training triples are edited and whose test triples measure post-edit plausibility.","marker":"He et al., 2024"},{"why":"Supplies the conceptualization and instantiation distillation procedure that ConKE adapts to enrich the edit targets.","marker":"Wang et al., 2024b"},{"why":"Supplies MEMIT, one of the editing algorithms used to write enriched knowledge into the LLM.","marker":"Meng et al., 2023"},{"why":"Supplies ROME, the second weight-editing algorithm used in the evaluation.","marker":"Meng et al., 2022"},{"why":"Supplies GRACE, the adapter-based editing method that provides an external-memory alternative.","marker":"Hartvigsen et al., 2023"},{"why":"Specifies the GPT-4o model used to prompt conceptualization and instantiation of the edited triples.","marker":"OpenAI, 2024b"},{"why":"Provides the zero-shot evaluation protocol and the five commonsense QA benchmarks used for downstream tests.","marker":"Ma et al., 2021a"}],"fun_headline_variants":["Few flagged beliefs, conceptualized, lift five commonsense QA benchmarks","Verifier flags bad beliefs; conceptualization generalizes fixes","ConceptEdit: small error set, conceptualized, boosts reasoning","From a handful of errors to broad commonsense gains via concepts","Conceptualization turns sparse edits into QA benchmark wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes VERA's plausibility scores are a trustworthy proxy for human commonsense, both when selecting which beliefs to edit and when judging whether the edits worked.","fun_headline_variants_meta":{"raw":{"variants":["Few flagged beliefs, conceptualized, lift five commonsense QA benchmarks","Verifier flags bad beliefs; conceptualization generalizes fixes","ConceptEdit: small error set, conceptualized, boosts reasoning","From a handful of errors to broad commonsense gains via concepts","Conceptualization turns sparse edits into QA benchmark wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000318,"raw_usage":{"total_tokens":1786,"prompt_tokens":921,"completion_tokens":865,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":781}},"tokens_in":537,"tokens_out":865,"duration_ms":7662,"temperature":1.0,"reasoning_tokens":781,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:57:01.567256+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent raters with measured inter-annotator agreement score a sample of the triples VERA flagged as implausible and a sample of post-edit generations, then compare per-item labels with VERA's scores; if agreement is near chance, or if editing the same number of randomly chosen triples produces the same downstream gains, the reported improvements are not attributable to VERA-guided conceptualized editing.","supporting_citations":[],"review_version":1}