{"id":"989bdebe-165e-48fd-878b-71c578fe846b","arxiv_id":"2501.00803","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Locating the encoder MLP and decoder cross-attention modules, then transferring edits between tasks by vector arithmetic, yields strong zero-shot event-relational reasoning on most of ten datasets.","lead":"This paper presents two techniques, ROLE and ABLE, that locate and edit specific internal modules of a language model to improve zero-shot event-relational reasoning across causal, temporal, and sub-event relations. The methods report state-of-the-art scores on most of ten benchmarks while using a fraction of the training time of prefix-tuning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's 'ABLE' row is the per-dataset best of four ABLE variants with an unreported alpha; no single fixed ABLE configuration attains the claimed SOTA on most datasets.","rationale":"The load-bearing condition for the abstract's SOTA claim is that a single ABLE procedure, without access to target labels, outperforms baselines on a majority of datasets. The Table 2 row fails this condition as written because it matches the per-dataset maximum of four ABLE variants, verified by exact F1 matches with Tables 3-5. No fixed variant reaches SOTA on more than 5 of 10 datasets, so 'most datasets' is true only under per-dataset oracle selection. This is an internal evaluation-protocol gap, not a disagreement with outside consensus. The reader's conditional verdict already flags the unreported alpha in Eq. 9, and I share that concern, but the stronger issue is the best-of-four selection; hence partial agreement. Independent support exists for the interpretability analysis and Table 8's computational-cost reduction, but neither supports the SOTA headline if the configuration is not fixed. I recommend keeping the CONDITIONAL verdict: the paper should be accepted only if the authors fix a configuration a priori, report alpha, and show that a prespecified variant retains SOTA on a majority of datasets.","tokens_in":25275,"tokens_out":13932,"duration_ms":131311,"concrete_test":"Recompute Table 2 with one prespecified ABLE configuration, e.g., ABLE2Dec with alpha=1, chosen before seeing any target labels, and count SOTA rows against the Table 2 baselines. If the count drops from the reported 7 to at most 5, the central claim rests on per-dataset best-of-four selection rather than on a fixed zero-shot method. The authors should also report the alpha value used for each original Table 2 row; if alpha was chosen using target labels, the zero-shot claim is invalid regardless of the F1 scores.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2's 'ABLE' row is not a single method. Cross-checking Table 2 against ablation Tables 3-5 shows every reported F1 is the maximum over the four ABLE variants: SCI-uni = ABLE2Dec, ESL-uni = ABLE1Enc, CTB-uni = ABLE2Dec, ESC-intra = ABLE1Enc, CTB-intra = ABLE2Dec, MAVEN-intra-causal = ABLE1Enc, ALT = ABLE2Dec, CNC = ABLE2Dec, MAVEN-intra-subevent = ABLE1Enc, HiEve = ABLE1Dec. The paper never states which variant Table 2 reports or how it was chosen, and Section 4.4 justifies analogies only 'based on the best results', i.e., after seeing test outcomes. Since the benchmark is presented as zero-shot, no validation split exists on which such a choice could legitimately be made. If the row is an oracle over four configurations, Table 2's 'SOTA on most datasets' is inflated by multiple comparisons: counting SOTA rows per fixed variant gives 3 (ABLE1Enc), 1 (ABLE2Enc), 3 (ABLE1Dec), and 5 (ABLE2Dec) of 10, whereas the reported per-dataset best obtains 7. Section 4.6's eigenvector similarity check does not test whether Eq. 9's composed edit improves a held-out target. Equation 9 compounds this: alpha is unreported, so the transfer rule is not reproducible and could encode target-task tuning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two model-editing methods for zero-shot event-relational reasoning with Flan-T5-large. ROLE locates key MLP/cross-attention modules via average indirect effects and computes parameter edits to improve reasoning; ABLE transfers edits between tasks using a parallelogram analogy in weight space (Eq. 8-9). The methods are evaluated on 10 datasets covering causal relation extraction/classification and sub-event extraction, with claims of state-of-the-art results and large computational savings.","tokens_in":25605,"tokens_out":6657,"duration_ms":57344,"significance":"If the claims were reproducible and internally consistent, the work would be valuable: it brings interpretable locate-and-edit techniques to reasoning tasks, reports a detailed module-level analysis, and demonstrates potential efficiency gains over prefix tuning. The paper also includes a thoughtful discussion of reasoning mechanisms and a novel analogy-based transfer idea. However, the central SOTA claim is currently undermined by the issues detailed below, so the significance is conditional on substantial revision.","major_comments":[{"comment":"The 'ABLE' row in Table 2 is not a single method. Cross-referencing the per-dataset F1 values with Tables 3-5 shows that the reported score is in each case the maximum over the four ABLE variants (ABLE1Enc, ABLE2Enc, ABLE1Dec, ABLE2Dec); for example, SCI-uni uses ABLE2Dec, ESL-uni uses ABLE1Enc, CTB-uni uses ABLE2Dec, and MAVEN-intra-subevent uses ABLE1Enc. The paper never states which variant is reported or how it was selected. Since the benchmark is presented as zero-shot, there is no validation split on which such a selection could legitimately be made. If the row is an oracle over four configurations, the 'SOTA on most datasets' claim is inflated by multiple comparisons. Report a fixed configuration or a principled selection rule, and adjust the SOTA claims accordingly.","section":"Section 4.3, Table 2 vs Tables 3-5"},{"comment":"The ABLE2Enc entry on HiEve reports P=59.00, R=98.82, F1=11.14. These three numbers are inconsistent: the harmonic mean of 59.00 and 98.82 is approximately 73.89, not 11.14. This indicates a transcription or computation error and casts doubt on the reliability of the other values in Table 5. The authors must correct the entry and re-verify all reported F1 scores.","section":"Table 5, ABLE2Enc row on HiEve"},{"comment":"The hyperparameter α in Eq. (9) is never reported, and the text does not state how it is chosen. The composed edit ΔW_D depends directly on α; without reporting its value or selection procedure, the method is not reproducible, and if α is tuned on target-task data, the zero-shot transfer claim would be compromised. Provide the α values used for each dataset and describe how they were selected.","section":"Section 3.2, Eq. (9)"},{"comment":"The analogicality analysis of editing magnitude is circular. Table 7 computes similarities between differences of edit vectors (e.g., ΔW_AB vs ΔW_CD) that are the same quantities used to motivate Eq. (9). This does not test whether the composed edit ΔW_D = ΔW_C − α(ΔW_A − ΔW_B) improves a held-out target. The only direct test is in the ablation tables, but those rely on the post-hoc oracle selection criticized above. A held-out evaluation of the composed edit is needed to validate the analogy.","section":"Section 4.6, Table 7"},{"comment":"The claim that 'ABLE 1Enc, ABLE 2Enc, ABLE 1Dec, and ABLE 2Dec improve the F1 score of all tasks' is false as stated: in Table 5, ABLE2Enc on HiEve has F1=11.14, which is lower than the w/oAll baseline of 11.18. This factual error should be corrected to reflect the per-dataset results.","section":"Section 4.4, last paragraph"},{"comment":"The paper uses 500 positive and 500 negative MAVEN samples for reasoning-oriented locating but never states that these samples are disjoint from the MAVEN-intra-causal and MAVEN-intra-subevent evaluation sets. Without this assurance, the zero-shot status of those two datasets in Table 2 is unclear. Specify the exact data split (e.g., train versus test) and confirm disjointness.","section":"Appendix A and Table 2"}],"minor_comments":[{"comment":"The token names are inconsistent: Table 1 uses '</s>' for the decoder cross-attention token, while Table 11's caption refers to the '<s>' token; please standardize the notation.","section":"Table 1 and Appendix B"},{"comment":"The conditioning notation in Eqs. (1) and (2) is ambiguous: it is unclear what x*, h*, and the patched module condition refer to, and why Eq. (1) conditions on x* in both terms while Eq. (2) does not. Clarify these definitions.","section":"Section 3.1.1, Eqs. (1)-(2)"},{"comment":"When discussing why ROLE performs poorly, the paper does not name the task; specify that this concerns causal relation classification and identify which datasets are affected.","section":"Section 4.4, first paragraph"},{"comment":"The column header 'MAVEN-intra' should be 'MAVEN-intra-causal' for consistency with Table 2 and Section 4.1.","section":"Table 3"},{"comment":"The list of baselines includes text-davinci-002 and text-davinci-003, but Table 2 reports only GPT-3.5 and GPT-4; clarify which models were actually evaluated on which datasets.","section":"Section 4.2 and Table 2"},{"comment":"The meanings of 'Params (M)' and 'Training time (s)' for ROLE and ABLE are not defined; clarify whether these refer to the number of edited parameters and the locating-plus-editing time, respectively.","section":"Section 4.7, Table 8"},{"comment":"The pseudocode has a formatting typo in the print statement, 'print i, count[i])', and needs a base case for i=0; also consider defining the eigenvalue threshold criterion more precisely.","section":"Appendix C, Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising direction, but the current version is not publishable as-is. The oracle selection in Table 2 and the impossible F1 in Table 5 are serious issues that directly affect the SOTA claim. The lack of reported α values and the circular analogy validation also need to be addressed. If the authors can provide a fixed configuration, report all hyperparameters, correct the data, and add a direct held-out evaluation of the composed edit, a revised version could be considered. I would also encourage the authors to clarify the MAVEN data split and to temper the abstract's 'SOTA' claim accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's headline result, \"ABLE achieves SOTA,\" is overstated as written. The ABLE row in Table 2 is the per-dataset best of four variants (ABLE1Enc/2Enc/1Dec/2Dec), and the paper never states this. No single fixed configuration achieves SOTA on most datasets; by my count one variant wins 3 datasets, another 5, and the other two 1-3. In a zero-shot benchmark there is no validation split that would make that selection legitimate, so the claim is inflated by post-hoc oracle selection. Also, alpha in Eq. 9 is never reported, so the transfer step is not reproducible.\n\nWhat is genuinely good: ROLE's localization analysis is cheap and interpretable, and the finding that encoder MLP and decoder cross-attention modules carry the relation-reasoning signal in Flan-T5-large is the kind of concrete mechanistic observation that is useful to the model-editing and event-reasoning communities. The overall cost numbers in Table 8 are striking. And the evaluation on the eight non-MAVEN datasets is genuinely zero-shot with respect to the target tasks—the edit vectors are built from MAVEN tasks A, B, C and then applied to new datasets.\n\nSoft spots beyond the oracle issue:\n- Table 5 contains an impossible triplet: ABLE2Enc on HiEve P=59.00, R=98.82, F1=11.14. The harmonic F1 is ~73.9. That looks like a typo, but it undermines trust in the table.\n- Section 4.4 says all four ABLE variants improve F1 on all tasks. As printed, ABLE2Enc on HiEve scores 11.14 vs. w/oAll 11.18, so that is false for at least one cell.\n- The analogy rule Eq. 9 is task-vector arithmetic. The paper doesn't cite the task-arithmetic literature, which would be the natural comparison. The similarity analysis in Section 4.6 tests the same weight differences used to define the analogy, so it is not strong external validation.\n- The 500+500 MAVEN locating samples may overlap with the MAVEN-intra evaluation subsets; the paper never states they are disjoint. That matters for the two MAVEN-intra rows.\n\nNone of this kills the central idea: ROLE/ABLE is a cheap, sub-second way to specialize a model for relation reasoning, and individual fixed ABLE variants do beat baseline T5 and UniEvent on several datasets. But the SOTA claim and the reproducibility need work before the results can be trusted.\n\nWho this is for: researchers working on model editing, parameter-efficient adaptation, and event-relation reasoning. It deserves a serious referee, not a desk reject, but the referee should demand a single fixed ABLE configuration, reported alpha/lambda, a corrected Table 5, and a statement about MAVEN overlap.","headline":"The SOTA claim is inflated by per-dataset cherry-picking across four ABLE variants, but the cheap editing-transfer idea and the localization findings are worth a serious look.","tokens_in":26215,"tokens_out":3778,"would_cite":false,"duration_ms":32700,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that locating one reasoning module and editing its weights transfers event-relation skills between tasks, beating prefix tuning and large closed models on most of ten zero-shot benchmarks.","keywords":["zero-shot learning","event-relational reasoning","knowledge editing","model interpretability","prefix tuning","analogy-based transfer","causal relation extraction","language model reasoning"],"falsifier":"Reproduce the ABLE transfer for a target task D without touching D's labels: compute $\\Delta W_C$, $\\Delta W_A$, and $\\Delta W_B$ from their MAVEN locating samples, sweep $\\alpha$ over a fine grid, and apply Equation 9 to the located module. If no single $\\alpha$ brings the transferred edit within one or two F1 points of the direct ROLE edit on a held-out subset of the same MAVEN family—or if the best $\\alpha$ changes sharply across the four task-quadruples in Table 6—the parallelogram premise fails. Also check whether any of the 500+500 locating samples appear in the MAVEN-intra evaluation subsets; if they do, the zero-shot numbers are inflated by memorization.","tokens_in":1878,"feed_emoji":"🧠","tokens_out":2714,"duration_ms":95897,"temperature":0.7,"pith_summary":"This paper tries to show that zero-shot event-relational reasoning—deciding whether events stand in causal, temporal, or sub-event relations—can be improved by locating and editing a small set of weights inside a large language model instead of training task-specific prefixes. The authors' ROLE method finds the single module (the encoder's MLP or the decoder's cross-attention) whose output most affects the yes/no answer, then edits its output projection weights with a constrained objective that suppresses the model's tendency to answer 'Yes' too often. Their ABLE method then treats the locating and editing information of three tasks as the corners of a parallelogram and computes the fourth task's edit by vector subtraction, claiming that this transfers reasoning knowledge across tasks. If the claim is right, state-of-the-art zero-shot results on most of ten event-relational benchmarks can be obtained in a fraction of a second of training, with an interpretable account of where the model stores relational cues.","feed_headline":"One edited layer outdoes prefix tuning in zero-shot event reasoning","feed_subtitle":"A locate-and-edit method transfers causal, temporal, and sub-event reasoning across tasks for a fraction of the cost.","key_machinery":"The load-bearing object is the edit vector $\\Delta W$ for a located module $H_{\\langle T,L \\rangle}$—the change to the output-projection weights of the encoder's MLP or the decoder's cross-attention. It is computed with a constrained objective (Equation 5, in the style of knowledge editing) using 500 positive and 500 negative MAVEN samples per task. The transfer identity is the parallelogram rule $\\Delta W_D = \\Delta W_C - \\alpha(\\Delta W_A - \\Delta W_B)$, with the location of D obtained by the same subtraction (Equation 8); the value of $\\alpha$ is not reported. The average indirect effect, Pearl's measure of how much a module changes the answer probability when its hidden state is perturbed, is what makes locating empirical: the key module is selected by its measured causal effect rather than by training.","core_discovery":"The paper's central claim is that a language model's ability to reason about event relations can be improved by locating one type of module—the MLP in the encoder and the cross-attention in the decoder of Flan-T5-large—and editing the weights of that module, rather than by training soft prompts. On the locating side, the paper uses the average indirect effect of each module on the yes/no answer to identify the token and layer that matter most, and it reports that the encoder MLP stores relational cues such as relation-type words and question words while the decoder cross-attention integrates those cues at the start token. On the transfer side, ABLE treats four tasks as the corners of a parallelogram: given located edits for three tasks A, B, C, the edit for a fourth task D is $\\Delta W_D = \\Delta W_C - \\alpha(\\Delta W_A - \\Delta W_B)$ and its location is obtained analogously. The paper reports state-of-the-art zero-shot F1 on most of ten datasets for causal and sub-event relation tasks, with the largest gains on sparse targets such as CTB-uni and MAVEN-intra sub-event extraction.","pith_inferences":["A reader should verify whether the 500+500 MAVEN locating samples are disjoint from the MAVEN-intra evaluation subsets; the paper never states this, and overlap would inflate the reported zero-shot gains through memorization.","If the parallelogram rule holds beyond this setting, the same vector arithmetic could be tested on other binary relational heads of a frozen transformer—contradiction, coreference, or sentiment-pair classification—where locating samples are abundant and target labels are scarce.","The paper's story that the edit mainly counters a 'Yes' bias suggests a sharper test: apply the same $\\Delta W$ to an unrelated binary task that also suffers from yes-hallucination; if it transfers, the edit vector is a bias correction, and if not, it is relation-specific.","The unreported $\\alpha$ is worth probing: reporting the sensitivity curve of F1 versus $\\alpha$ for each task quadruple would show whether the analogy has one universal strength or needs per-pair tuning."],"forward_implications":["Zero-shot event-relational reasoning can be improved without training prefixes at all: editing the output weights of one located module in a frozen Flan-T5-large suffices.","Because the transfer rule is arithmetic on edit vectors, a new task D costs only the cheap computations of the A, B, C edits; the paper reports ABLE training times of 0.09 seconds (encoder) and 0.03 seconds (decoder) on CTB-uni.","The located modules give a mechanism story: the encoder MLP handles relation-type and question words, while the decoder cross-attention routes relational information from the start token, which also explains the model's 'Yes' bias and where to correct it.","The analogy is not uniform: causal/temporal and causal/sub-event pairs transfer well, while temporal/sub-event transfer is weak, so the method's gains should be expected on tasks near one of those axes.","On the weakest baselines the margins are large (for example, CTB-uni 13.64 F1 versus 8.95 for UniEvent and 2.49 for GPT-4), indicating that the edit also fixes a systematic over-prediction of 'Yes'."],"supporting_citations":[{"why":"Supplies UniEvent, the multi-task prefix-tuning baseline ABLE must beat, and the evaluation setup for SCI-uni, ESL-uni, and CTB-uni.","marker":"Tao et al., 2023"},{"why":"Supplies the locate-and-edit weight-update rule (Equation 5) that ROLE adapts to compute each task's editing magnitude.","marker":"Meng et al., 2022b"},{"why":"Supplies the average indirect effect measure that defines how ROLE locates the key module at a token and layer.","marker":"Pearl, 2022"},{"why":"Provides the ESC-intra, CTB-intra, and MAVEN-intra-causal evaluation setting and the observation that LLMs over-predict causal relations.","marker":"Gao et al., 2023"},{"why":"Supplies the T5 architecture and the C4 pre-training corpus used to compute the regularizer $C_0$ in the editing objective.","marker":"Raffel et al., 2020"},{"why":"Supplies the MAVEN-ERE dataset used both for the 500+500 locating samples per task and for the MAVEN-intra evaluation subsets.","marker":"Wang et al., 2022"},{"why":"Supplies the EventStoryLine corpus that defines the ESL-uni and ESC-intra causal relation extraction benchmarks.","marker":"Caselli and Vossen, 2017"},{"why":"Supplies the Causal News Corpus used as the CNC causal relation classification benchmark.","marker":"Tan et al., 2022"}],"fun_headline_variants":["Edit key modules, not prefixes, for zero-shot event reasoning","Analogy-based module edits outdo prefix tuning in zero-shot event reasoning","Locate and edit key modules for efficient zero-shot event-relational reasoning","One module edit replaces prefix tuning for zero-shot event reasoning"],"cache_read_input_tokens":28160,"weakest_assumption_plain":"The load-bearing premise is that edit vectors compose by parallelogram arithmetic in weight space with a stable $\\alpha$; the paper also never states that the 500+500 MAVEN locating samples are disjoint from the MAVEN-intra test subsets.","fun_headline_variants_meta":{"raw":{"variants":["Edit key modules, not prefixes, for zero-shot event reasoning","Analogy-based module edits outdo prefix tuning in zero-shot event reasoning","Locate and edit key modules for efficient zero-shot event-relational reasoning","One module edit replaces prefix tuning for zero-shot event reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000879,"raw_usage":{"total_tokens":3798,"prompt_tokens":941,"completion_tokens":2857,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":2783}},"tokens_in":557,"tokens_out":2857,"duration_ms":21778,"temperature":1.0,"reasoning_tokens":2783,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:43:59.830906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the ABLE transfer for a target task D without touching D's labels: compute $\\Delta W_C$, $\\Delta W_A$, and $\\Delta W_B$ from their MAVEN locating samples, sweep $\\alpha$ over a fine grid, and apply Equation 9 to the located module. If no single $\\alpha$ brings the transferred edit within one or two F1 points of the direct ROLE edit on a held-out subset of the same MAVEN family—or if the best $\\alpha$ changes sharply across the four task-quadruples in Table 6—the parallelogram premise fails. Also check whether any of the 500+500 locating samples appear in the MAVEN-intra evaluation subsets; if they do, the zero-shot numbers are inflated by memorization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies UniEvent, the multi-task prefix-tuning baseline ABLE must beat, and the evaluation setup for SCI-uni, ESL-uni, and CTB-uni."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the average indirect effect measure that defines how ROLE locates the key module at a token and layer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ESC-intra, CTB-intra, and MAVEN-intra-causal evaluation setting and the observation that LLMs over-predict causal relations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the EventStoryLine corpus that defines the ESL-uni and ESC-intra causal relation extraction benchmarks."}],"review_version":1}