{"id":"492169f4-96c6-40ab-83ed-9aa60a12e6ce","arxiv_id":"2412.17727","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EditCoT edits a language model's chain-of-thought to inject new knowledge, using a trained CoT editor and iterative conflict detection, and reports competitive results across three benchmarks.","lead":"EditCoT updates a large language model's knowledge without retraining by generating a chain-of-thought, checking each reasoning step against new knowledge, and rewriting conflicting steps with a small specialized editor. It reports strong results on three knowledge-editing benchmarks in English and Chinese, though one existing method still surpasses it on the main multi-hop QA test.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's unqualified 'state-of-the-art' claim is contradicted by the paper's own Table 1 and Table 3, where RAE beats EditCoT on MQuAKE-CF-3k/Llama (54.1 vs 35.4) and on GPT-4o (59.7 vs 45.0).","rationale":"The reader's conditional verdict is sensible. We focus on a different load-bearing element than the stated weakest assumption: the abstract's unqualified 'state-of-the-art' claim is directly contradicted by the paper's own Table 1 and Table 3, independent of any speculation about the conflict detector. The paper's defense that it is best on 6/8 results depends on excluding RAE from datasets where RAE is 'not applicable'; on the shared benchmark, RAE is superior. This is an internal inconsistency, not an external consensus disagreement. That said, the reader's concern about conflict detection is also valid and was explicitly demonstrated in Table 9; we simply find the SOTA overclaim to be more decisive because even a perfect conflict detector would not make the SOTA claim true on MQuAKE with Llama or GPT-4o. The paper has genuine strengths: the training-data construction is fully described, the leakage check (Appendix H) is a thoughtful control, and the method does generalize to DUNE and LeKUBE where RAE cannot run. The correct fix is to temper the claims and complete the baseline table with Verify-and-Edit; the research contribution remains publishable as a conditional accept.","tokens_in":21734,"tokens_out":10591,"duration_ms":90862,"concrete_test":"Recompute the GPT-4o comparison on the same 300 randomly sampled MQuAKE-CF-3k queries with a fixed random seed and the same Llama-3-8B proxy/editor, and also tabulate the full MQuAKE-CF-3k results for EditCoT and RAE under the identical retrieval setting. If RAE's accuracy remains above EditCoT's in both, the abstract's SOTA claim is false and should be revised to 'competitive/state-of-the-art among task-agnostic methods'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract) is that EditCoT 'achieves state-of-the-art performance while offering superior generalization, effectiveness, and stability compared to existing methods.' This is not supported by the paper's own tables. On MQuAKE-CF-3k with Llama-3-8B-Instruct, RAE scores 54.1% vs EditCoT's 35.4% (Table 1); on the GPT-4o evaluation, RAE scores 59.7% vs EditCoT's 45.0% (Table 3). The paper's 'top performance on 6 out of 8' (Section 5.1) is achieved only because RAE is declared 'not applicable' on DUNE and LeKUBE, so on the one primary benchmark where a direct comparison is possible, EditCoT is not SOTA. Thus the unqualified 'state-of-the-art' claim in the Abstract is internally inconsistent with the reported results. The framework may be a useful competitive baseline, but the headline claim needs to be qualified (e.g., 'best among methods applicable to all three benchmarks' or 'competitive with RAE on MQuAKE while extending to new domains').","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EditCoT, a non-parametric knowledge-editing framework that updates an LLM by iteratively revising its chain-of-thought. The method first generates an initial CoT and answer, then repeatedly retrieves updated knowledge, detects conflicts between the CoT and the retrieved knowledge using a three-way token-probability classifier, and revises the CoT with a separately trained CoT editor. The editor is trained on synthetically constructed data derived from HotpotQA without human annotation. The authors evaluate EditCoT on MQuAKE-CF-3k, DUNE, and LeKUBE using Llama-3-8B-Instruct, Qwen2.5-14B-Instruct, and GPT-4o, reporting strong performance on DUNE and LeKUBE, competitive results on MQuAKE-CF-3k with Qwen, and superior locality, stability across batch sizes, and inference efficiency relative to several baselines. The paper also includes ablations showing the importance of conflict detection and the trained editor, plus a data-leakage check.","tokens_in":21992,"tokens_out":5645,"duration_ms":49782,"significance":"If the results hold, EditCoT is a meaningful step toward task-agnostic in-context knowledge editing: it does not require retraining at edit time, works across languages and task types, and is applicable to black-box models. The release of code and data, the leakage check in Appendix H, and the component ablations in Table 5 are concrete strengths that increase confidence in the empirical claims. The main weakness is that the headline 'state-of-the-art' claim is not supported by the paper's own comparative tables, and the core conflict-detection module is not directly evaluated. The approach remains a competitive and generalizable baseline that advances the editing-by-reasoning direction, provided the claims are appropriately qualified.","major_comments":[{"comment":"The unqualified claim that EditCoT 'achieves state-of-the-art performance' is internally inconsistent with the reported results. On MQuAKE-CF-3k with Llama-3-8B-Instruct, RAE reaches 54.1% while EditCoT reaches 35.4% (Table 1); on the GPT-4o evaluation, RAE reaches 59.7% while EditCoT reaches 45.0% (Table 3). The 'top performance on 6 out of 8 results' in Section 5.1 is obtained only because RAE is marked not applicable on DUNE and LeKUBE. On the one primary benchmark where a direct comparison is possible, EditCoT is not state of the art. The abstract, introduction, and conclusion should qualify the claim (e.g., 'best among methods applicable to all three benchmarks' or 'competitive with RAE on MQuAKE while extending to new domains').","section":"Abstract and Section 5.1, Tables 1 and 3"},{"comment":"The synthetic data construction assumes that 'the divergence occurs only once in most of the cases' between the RAG-generated CoT and the prefix-guided CoT. This assumption is never validated quantitatively, and it is load-bearing for the correctness of the training tuples: if the two CoTs diverge at multiple steps, then the constructed tuple (Q, CoT_k, rewrite(Step_k), CoT) may not represent a coherent edit, and the editor could learn spurious mappings. The paper should report the proportion of accepted examples where the divergence is indeed unique, and, if multiple divergences occur, analyze whether excluding or transforming them changes the editor's performance on the downstream benchmarks.","section":"Section 3.3.2, Part III"},{"comment":"The conflict-detection module is a core component: removing it in Table 5 causes a large drop on MQuAKE-CF (from 40.3 to 32.3 for Llama and from 40.3 to 33.0 for Qwen) and on LeKUBE MCQ (from 58.3 to 48.9). Yet the paper never directly measures the module's precision or recall on the evaluation benchmarks. The failure case in Table 9 shows a false conflict leading to an incorrect edit, and the paper only gives an anecdotal estimate of '1-2 cases out of 200 manually inspected examples.' A systematic evaluation of the conflict detector (e.g., accuracy on held-out conflict/support/unrelated labels sampled from the test sets) would substantiate the claim that the iterative editing loop is reliable and would give readers a concrete sense of when the method is likely to fail.","section":"Section 3.2 Step 2 and Table 9"}],"minor_comments":[{"comment":"The conflict-detection threshold of 0.6 is a hand-set hyperparameter, but no sensitivity analysis is provided. A brief plot or table showing performance as the threshold varies would help establish robustness.","section":"Section 3.4"},{"comment":"In the conflict-detection prompt, the few-shot assistant responses include both the relevance label and a 'New Thoughts:' continuation. Since the model is supposed to output only the label at test time, the presence of new-thought examples may bias the classifier. Consider removing the 'New Thoughts' portion from the few-shot classification examples or verifying that the probability-based classification is unaffected.","section":"Appendix J.2"},{"comment":"Figure 4 is split due to a wide horizontal span, which makes it hard to compare all methods. Using a log-scale x-axis or separate subplots with a shared legend would improve readability.","section":"Figure 4"},{"comment":"The failure-case description states that the first error is highlighted in red, but the text in the table is not visibly colored. Please use a printer-friendly marker such as bold type or an asterisk.","section":"Table 9"},{"comment":"The limitations section mentions higher GPU resource consumption but does not report actual memory or compute numbers. A short quantitative comparison (e.g., peak GPU memory for EditCoT vs. Mello/PokeMQA) would make the trade-off concrete.","section":"Section 7, Limitations"},{"comment":"LeKUBE is cited as both Wang et al. 2024a and Wang et al. 2024b in the text; please disambiguate these references and use a consistent citation style throughout.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a solid empirical contribution to in-context knowledge editing, and the code/data release plus the leakage control are commendable. The main issue is the overstated 'state-of-the-art' claim in the abstract and conclusion, which is contradicted by the paper's own Tables 1 and 3; this is fixable by rewording. The undeveloped evaluation of the conflict-detection module is also worth addressing before publication. I do not see grounds for rejection: the method is novel, the ablations are informative, and the limitations are honestly discussed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EditCoT is worth knowing about. The core contribution—a dedicated trained CoT editor plus explicit three-way conflict detection—is genuinely new relative to Verify-and-Edit, which relies on self-consistency and in-context prompting. The paper also does several things right: the training-data construction from HotpotQA without human annotation is clever, the leakage check in Appendix H is reassuring, the component ablations in Table 5 show both pieces matter, and the failure cases in Tables 9-10 are reported honestly. They also ship code and data.\n\nThe soft spots are real but mostly fixable. The abstract's 'state-of-the-art' claim is not supported by the paper's own numbers. On MQuAKE-CF-3k with Llama-3-8B, RAE gets 54.1 vs EditCoT's 35.4; on GPT-4o, RAE gets 59.7 vs 45.0. The 'top on 6 of 8' only holds because RAE is marked not applicable on DUNE and LeKUBE. On the one benchmark with a direct head-to-head, EditCoT is not SOTA. That needs rephrasing—'competitive and more general' or 'best among methods applicable to all three benchmarks' would be accurate.\n\nMissing the closest prior baseline from the main tables is a second issue. Verify-and-Edit appears only in Appendix B, with a partial comparison. Given the paper's title and framing, that baseline belongs in the main experiments, even if it needs adaptation. Also, there are no error bars anywhere, and the hand-set 0.6 conflict threshold is a free parameter. Those are minor but worth tightening.\n\nThe central mechanism—iterative CoT editing with a trained editor—holds up well enough. The ablations show it does real work, and the leakage check addresses the obvious worry that the editor memorized benchmark answers. The failure case in Table 9 shows the classifier can misfire and cause a wrong edit, but the authors acknowledge it and it appears rare.\n\nWho is this for? Researchers in knowledge editing and in-context learning. It is a solid, well-described method that deserves referee time, but it needs a major revision to temper the headline and complete the baseline comparison. I would accept it for peer review and hope the reviewers push on the SOTA claim.","headline":"A genuinely useful CoT-editing framework with honest ablations, undermined by an overclaimed SOTA headline that its own tables refute.","tokens_in":22554,"tokens_out":2029,"would_cite":true,"duration_ms":18604,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EditCoT rewrites a model's chain-of-thought to update its knowledge without retraining.","keywords":["knowledge editing","in-context editing","chain-of-thought","iterative reasoning refinement","multi-hop question answering","legal knowledge update","black-box LLMs","conflict detection"],"falsifier":"Run EditCoT on MQuAKE-CF-3k with a conflict detector replaced by an oracle that knows ground-truth conflicts, and compare with the model-based detector; if accuracy does not improve, conflict detection is not the mechanism the paper claims. The mirror test is to inject a fixed false conflict on unrelated retrieved facts and count how often the final answer flips; the paper's Table 9 already shows one such flip, so a dataset where this happens frequently would refute the stability claim.","tokens_in":21491,"feed_emoji":"🧠","tokens_out":6242,"duration_ms":53413,"temperature":0.7,"pith_summary":"EditCoT is a knowledge-editing framework that claims to update an LLM's factual knowledge by rewriting the model's chain of thought rather than retraining or changing weights. The method has the model answer a question, generate a step-by-step reasoning chain, retrieve updated facts, detect which steps conflict with those facts, and iteratively hand conflicting chains to a separately trained editor that produces a corrected chain. The paper reports that this one pipeline, with no task-specific retuning, reaches the best or second-best accuracy on multi-hop triple-based QA, free-form scientific and arithmetic edits, and Chinese legal statute updates, in both English and Chinese. A sympathetic reader would care because it points toward a way to keep deployed models current without expensive retraining, while preserving the model's original abilities.","feed_headline":"Chain-of-thought editing updates LLM knowledge without retraining","feed_subtitle":"One pipeline handles multi-hop QA, free-form edits, Chinese legal changes, and black-box models like GPT-4o.","key_machinery":"The load-bearing mechanism is the iterative edit loop built around a small, task-agnostic CoT editor. The editor is trained once, on automatically generated examples derived from HotpotQA questions, to take a question, an old chain, and new knowledge and output a revised chain; no retraining is needed when the knowledge changes. In inference it is driven by a three-way conflict classifier (conflict, support, unrelated) that decides whether the current reasoning step disagrees with retrieved updated facts, and by a retriever that maps old facts to their replacements or finds post-edit knowledge directly.","core_discovery":"The central discovery claimed is that knowledge conflicts can be resolved at the level of the reasoning trace. EditCoT treats a chain-of-thought as an editable object: the unedited LLM generates the chain and classifies each retrieved fact as support, conflict, or unrelated; on a conflict, a trained CoT editor rewrites the chain; the loop repeats until no conflict remains; and the edited chain then drives the final answer. The paper reports state-of-the-art or near-state-of-the-art accuracy across MQuAKE-CF-3k, DUNE, and LeKUBE on Llama-3-8B-Instruct and Qwen2.5-14B-Instruct, plus competitive results on GPT-4o using an 8B editor as a proxy, and shows that removing either conflict detection or the trained editor degrades performance.","pith_inferences":["The paper leaves implicit that the bottleneck may shift from editing to detection: its own failure analysis shows a false conflict can corrupt an otherwise correct chain, so better retrieval and conflict classification could improve EditCoT more than a stronger editor.","One extension beyond the paper is to use the same separation of reasoning path and fact memory to correct other reasoning flaws, not just factual updates, such as outdated procedures or stale instructions.","A testable extension would vary the retriever quality while keeping the editor fixed; if gains track retrieval quality more than editor size, that tells practitioners where to invest.","The method's dependence on a separately trained editor suggests a modular deployment where a small editor is shared across many large backbones, a configuration the paper only partially explores."],"forward_implications":["If the results hold, the same EditCoT pipeline transfers across task types and languages without prompts being redesigned, covering triple-based multi-hop QA, free-form reasoning updates, and legal statute changes.","Knowledge updates would no longer require retraining or weight modification, and the editor can be much smaller than the backbone model, so continuous updates become feasible on modest hardware.","Because the method needs no internal model access, it extends to proprietary black-box models such as GPT-4o, with a smaller open model acting as editor.","Robustness results suggest the method preserves unrelated knowledge better than baselines and degrades less as batch size grows, which matters for realistic bulk edits."],"supporting_citations":[{"why":"Supplies the MQuAKE-CF-3k benchmark and the Mello baseline that EditCoT must beat on multi-hop triple-based QA.","marker":"Zhong et al., 2023"},{"why":"PokeMQA, the strongest in-context baseline for MQuAKE, whose scope-detection design EditCoT compares against.","marker":"Gu et al., 2024a"},{"why":"RAE, the knowledge-graph in-context editor that EditCoT challenges on MQuAKE and whose graph requirement limits generalization.","marker":"Shi et al., 2024b"},{"why":"DUNE benchmark provides free-form edit tasks in arithmetic, scientific reasoning, and new-information settings.","marker":"Akyürek et al., 2023"},{"why":"LeKUBE legal benchmark supplies the Chinese vertical-domain knowledge-update tasks.","marker":"Wang et al., 2024b"},{"why":"HotpotQA provides the human questions from which the CoT editor's training data is automatically built.","marker":"Yang et al., 2018"},{"why":"Meta-Llama-3-8B-Instruct serves as one backbone and as the proxy editor in GPT-4o experiments.","marker":"Dubey et al., 2024"},{"why":"Qwen2.5-14B-Instruct serves as the other backbone, including Chinese-language LeKUBE evaluations.","marker":"Team, 2024"},{"why":"GPT-4o is the black-box model used to test whether EditCoT works without white-box access.","marker":"OpenAI et al., 2023"}],"fun_headline_variants":["Edit LLM knowledge by editing the reasoning chain itself","No retraining needed: Edit chain-of-thought to update LLM facts","CoT editing updates LLM knowledge across tasks and languages","EditCoT rewrites reasoning traces to update LLM knowledge","Update LLM knowledge by editing the chain-of-thought"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the unedited LLM can reliably classify retrieved facts as supporting, contradicting, or unrelated to each reasoning step; if the classifier mistakes an unrelated fact for a contradiction, the editor rewrites a correct chain and the final answer can become wrong, as the paper's Table 9 shows.","fun_headline_variants_meta":{"raw":{"variants":["Edit LLM knowledge by editing the reasoning chain itself","No retraining needed: Edit chain-of-thought to update LLM facts","CoT editing updates LLM knowledge across tasks and languages","EditCoT rewrites reasoning traces to update LLM knowledge","Update LLM knowledge by editing the chain-of-thought"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000655,"raw_usage":{"total_tokens":3005,"prompt_tokens":958,"completion_tokens":2047,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":1962}},"tokens_in":574,"tokens_out":2047,"duration_ms":20579,"temperature":1.0,"reasoning_tokens":1962,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:14:11.493334+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EditCoT on MQuAKE-CF-3k with a conflict detector replaced by an oracle that knows ground-truth conflicts, and compare with the model-based detector; if accuracy does not improve, conflict detection is not the mechanism the paper claims. The mirror test is to inject a fixed false conflict on unrelated retrieved facts and count how often the final answer flips; the paper's Table 9 already shows one such flip, so a dataset where this happens frequently would refute the stability claim.","supporting_citations":[],"review_version":1}