{"id":"4d076711-3920-40df-84dd-9e1d012584b0","arxiv_id":"2502.03954","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"MAQInstruct reformulates event relation extraction as relation-specific multiple-answer QA with a bipartite matching loss, reducing inference samples from n^2 to k*n and improving F1 across three LLMs on four datasets.","lead":"MAQInstruct reframes event relation extraction as a multiple-answer question answering task, asking a language model to list events that share a specific relation with a given event, instead of classifying each event pair. It also uses a bipartite matching loss to make the model insensitive to the order of the generated answers, cutting inference cost and improving accuracy on four event relation datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The order-invariance claim covers only the multiple-answer list; the dependency parsing chain, whose removal causes the largest ablation drop, is still an ordered generated sequence, so the sequence-dependency problem is displaced, not eliminated.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the bipartite matching loss does not cover the dependency parsing chain, and the ablation shows the DPC is the most influential component. My stress-test agrees. The main claims that would have to be true for the paper's central contribution to hold are (1) that the multiple-answer reformulation reduces inference samples as described, and (2) that the bipartite matching loss reduces dependence on generation order. The first claim is supported by the sample-count comparison in Table 2 and by the construction logic. The second claim is only partially supported: the BPM loss demonstrably stabilizes the multiple-answer ordering (Section 3.4), but the DPC is an ordered generated sequence that the paper neither makes order-invariant nor tests for order sensitivity. Removing the DPC is the largest ablation drop, so the model's performance is tied to a sequence that is not covered by the proposed remedy. This is a real gap in the paper's argument, but it does not overturn the empirical F1 improvements over InstructERE and BertERE, nor does it undermine the sample-reduction contribution. The paper reports averages over five seeds (Appendix B), which is positive, yet no error bars or significance tests are provided, so the robustness of the F1 differences is uncertain. The lack of code and the absence of a citation for InstructERE further complicate verification, but these are secondary to the technical gap. Given the conditional verdict already acknowledges these limitations, my read does not change the verdict. The concrete test would settle whether the DPC order actually matters; until then, the sequence-dependence claim should be treated as conditional.","tokens_in":11024,"tokens_out":3232,"duration_ms":32770,"concrete_test":"Retrain MAQInstruct on MECI (or a subset of MATRES) with the target DPC edges presented in several orders: (a) the paper's fixed order, (b) random shuffle per instance, (c) reverse order, and (d) alternative tie-breaking from the dependency parser. Evaluate on the same test splits. If test F1 varies by more than ~1 point across orders, the model remains order-dependent and the claimed reduction in sequence dependency is not achieved for the DPC component. Alternatively, with a fixed trained model, shuffle the gold DPC in the decoder input at inference time (if the DPC is part of the prompt) and compare answer F1; sensitivity would confirm the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution (abstract, Section 2.2, Section 3.4) is that the bipartite matching loss 'reduces the dependency of the instruction-based method on the generation sequence.' However, the loss in Eq. 4 applies CE to the dependency parsing chain (DPC) and BPM only to the multiple-answer portion. The DPC is itself a generated, order-sensitive sequence: Figure 1 shows a fixed ordering of dependency edges, and Appendix A specifies tie-breaking by appearance order. Table 3 shows that removing the DPC causes the largest performance drop (MATRES F1 84.7→81.4, MECI 62.3→58.4). Thus the model's overall accuracy depends heavily on a generated sequence whose order is not addressed by the claimed remedy. The BPM loss only reorders the final answer list; it cannot fix errors arising from an arbitrary or suboptimal DPC order. The paper does not test alternative DPC orders (e.g., shuffled, reversed, or parser-determined vs. alternative tie-breaking). Consequently, the claim that sequence dependence is reduced is only partially established: the problem may have been displaced onto the DPC rather than eliminated. The case study (Appendix E) even shows errors attributed to 'the complexity of the dependency parsing chain,' indicating the DPC is not a stable, order-free representation. This concern does not invalidate the reported F1 gains, but it does undermine the generality of the order-invariance claim and the claimed advantage over InstructERE on that specific issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MAQInstruct, an instruction-tuned framework for unified event relation extraction. It replaces pairwise event-event questions with relation-specific instructions that ask the model to select one or more event mentions from the context, reducing the number of inference queries from quadratic to linear in the number of mentions per document. It also introduces a bipartite matching loss (BPM) applied to the multiple-answer portion of the generated output, intended to make the model insensitive to the order in which answers are generated. Experiments on MAVEN-ERE compare MAQInstruct with BertERE and an InstructERE baseline across three LLMs (ChatGLM3, Qwen, Llama2), and cross-corpus/zero-shot results are reported on HiEve, MATRES, and MECI. The paper reports consistent F1 improvements over InstructERE (about 3.9-4.2 points on the four MAVEN-ERE subtasks for Llama2) and an inference-time reduction from 813 to 25 minutes for coreference on an A100-80G setup.","tokens_in":11275,"tokens_out":8950,"duration_ms":82971,"significance":"The empirical scope is a genuine strength: the method is evaluated on four ERE datasets, three LLM backbones, and five seeds, and the inference-cost comparison in Table 2 is concrete and central. The main efficiency claim (quadratic to linear query count) is well motivated and supported by the reported costs. If the order-invariance claim were fully established, the BPM component would be a useful and transferable idea. However, the present evidence supports order-invariance only for the final answer list, not for the full generated output, which weakens the contribution's stated scope.","major_comments":[{"comment":"The claim that MAQInstruct reduces the dependency on the generation sequence is only partially supported. The BPM loss in Eq. (4) is applied only to the multiple-answer part; the dependency parsing chain (DPC) is generated with ordinary cross-entropy under a fixed order (Appendix A), and Table 3 shows that removing the DPC causes the largest performance drop (MATRES F1 84.7 to 81.4; MECI F1 62.3 to 58.4). No experiment varies the DPC order, although Appendix E itself attributes errors to 'the complexity of the dependency parsing chain.' The paper should either restrict the order-invariance claim to the multiple-answer list or demonstrate that the DPC is order-stable (e.g., by comparing parser-determined, shuffled, and reversed DPC orders).","section":"§2.2, Eq. (4); §3.6, Table 3"},{"comment":"The comparison against BertERE is presented as a positive result, but the gains are as small as 0.4 and at most 1.7 F1 points, and no standard deviations or significance tests are reported even though Appendix B states the results are averaged over five random seeds. Without error bars or a paired test, the claim that MAQInstruct 'enhances' over BertERE is not statistically supported; at minimum, report per-seed variance or a significance test for the four MAVEN-ERE subtasks.","section":"Table 1; Appendix B"},{"comment":"The formal definition of the matching loss is ambiguous. N2 is introduced as the length of the multiple-answer part in tokens, but the permutation Ψ_N2 in Eqs. (2)-(3) is over token positions, with c_i a target vocabulary id; for multi-token answers, matching individual tokens is not equivalent to matching whole answer spans. Please state clearly whether the Hungarian algorithm operates over answer spans or tokens, and how variable-length answers are handled; this is necessary to reproduce the loss exactly.","section":"§2.2, Eqs. (2)-(3)"}],"minor_comments":[{"comment":"InstructERE is never cited or referenced; please either cite the original work or state explicitly that it is an internal baseline, and if it is based on the authors' prior ChatUIE [37], say so.","section":"Section 3.1"},{"comment":"The claimed '4.9%' improvement for coreference over InstructERE does not match any of the four coreference metrics in Table 1 (the closest is 4.2 points on BLANC); please correct the number.","section":"Section 3.2"},{"comment":"The plot has no axis labels or legend entries explaining what is being measured; please add labels for F1 and for the dataset/sequence-condition grouping.","section":"Figure 3"},{"comment":"The term 'zero-shot' is used for models trained on MAVEN-ERE and evaluated on HiEve/MATRES/MECI; this is cross-corpus transfer rather than zero-shot learning in the usual sense, so the terminology should be clarified.","section":"Section 3.5"},{"comment":"The n×n sample count should specify that n is the number of event mentions per document rather than the total number of mentions in the dataset; otherwise the reported 631,486 coreference queries for InstructERE appear inconsistent with the 17,780 total mentions.","section":"Table 2; Section 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the uncited InstructERE baseline is the main attribution concern; the authors have a closely related prior paper (ChatUIE, ref. [37]) and should clarify whether InstructERE is that system or a separate baseline. The paper is a compact WWW Companion contribution; the claims are somewhat overstated relative to the evidence, but the core efficiency idea is sound and the experiments are substantial enough for a major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I've read MAQInstruct. Main take: it's a genuinely useful empirical result for instruction-based event relation extraction, but the headline claim about reducing sequence dependence only holds for the final answer list, not for the dependency parsing chain they use as an intermediate representation.\n\nWhat's new: the multiple-answer QA formulation, where you query each relation type with one event mention and ask the model to select other mentions, cuts inference samples from n^2 to k*n. Their Table 2 shows the concrete effect: for coreference on MAVEN-ERE, queries drop from 631,486 to 17,780 and wall-clock from 813 to 25 minutes on the same hardware. That's a real, mechanistically explained cost reduction, and they show consistent F1 gains over InstructERE across three LLMs. They also demonstrate that the bipartite matching loss makes the model robust to the ordering of the multiple-answer list—they shuffle, reverse, sort by distance, and the BPM-trained model is flat. That part is clean and the ablation supports it.\n\nThe soft spots are about what the claim covers. The BPM loss applies only to the multiple-answer list. The dependency parsing chain—the string of dependency edges that precedes the answers—is itself generated in a fixed order, and the ablation in Table 3 shows that removing the DPC causes the largest drop (MATRES 84.7→81.4, MECI 62.3→58.4). They never test alternative DPC orderings. So the sequence-dependence problem is displaced, not eliminated; the model still depends on getting an ordered DPC right. Their own case study in Appendix E shows errors caused by the complexity of the DPC. That doesn't invalidate the F1 gains, but it does undermine the generality of the order-invariance claim.\n\nOther issues: InstructERE is not cited anywhere, which is odd given it's their main baseline—if it's their own prior work, the self-citation is missing; if it's someone else's, that's a citation gap. There are no error bars or significance tests despite averaging five seeds, and the gains over BertERE are small (0.4–1.7 points). No code release either.\n\nOverall, the paper is worth engaging with. The efficiency result is solid, and the BPM robustness study is informative. The order-invariance claim needs qualification, and the missing baseline citation needs fixing. For a venue like WWW Companion, this is acceptable with minor revisions; if it were a journal submission, I'd ask for variance estimates, alternative DPC orderings, and a proper citation for InstructERE.\n\nI'd send it to a serious referee, but tell them to focus on the DPC issue.","headline":"Solid efficiency win for instruction-based ERE, but the order-invariance claim only covers the answer list, not the dependency chain that actually drives performance.","tokens_in":11872,"tokens_out":3589,"would_cite":true,"duration_ms":29541,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MAQInstruct recasts event-relation extraction as multiple-answer selection among marked event mentions, using a bipartite matching loss to remove answer-order effects and cutting inference queries from $n^2$ to $k \\times n$.","keywords":["event relation extraction","instruction tuning","large language models","multiple-answer question answering","bipartite matching loss","dependency parsing chain","event coreference","temporal relation extraction"],"falsifier":"Shuffle the order of the edges in the dependency parsing chain at inference time (for example, reverse the chain, or sort edges randomly) while keeping the same answers; if MATRES or MECI F1 drops materially below the reported numbers, the method still depends on generation order, contradicting the claim that the bipartite matching loss removes sequence sensitivity. Alternatively, replace the generated chain with an unordered set of dependency edges; if accuracy collapses, the chain is functioning as a fixed-order crutch rather than a content scaffold.","tokens_in":10774,"feed_emoji":"🔗","tokens_out":11776,"duration_ms":99037,"temperature":0.7,"pith_summary":"The paper's central claim is that instruction-based event relation extraction fails for two fixable reasons: it asks one question per event pair (so $n^2$ samples per document), and autoregressive generation makes the answer order matter even though event relations are not sequential. MAQInstruct fixes both by inverting the instruction: given a relation, the model selects which marked event mentions in the context satisfy it, cutting the sample count to $k \\times n$ where $k$ is the number of relation types. A bipartite matching loss treats the selected answers as an unordered set, so the model is not penalized for producing them in any order. If right, the same framework handles coreference, temporal, causal, and sub-event relations with one instruction-tuned LLM, and on MAVEN-ERE it beats both the pairwise instruction baseline and a classification baseline while reducing coreference inference queries from 631,486 to 17,780 and wall-clock time from about 813 to 25 minutes.","feed_headline":"Event-relation extraction: 35x fewer queries, better F1","feed_subtitle":"It asks which events fit each relation, cutting queries 35x while beating classification and instruction baselines.","key_machinery":"The central machinery is the multiple-answer question format plus a bipartite matching loss. Each event mention in the context is preceded by a distinct marker token (for example <0x85>), and an instruction names one relation and one query mention; the model must output a dependency parsing chain—a sequence of typed syntactic edges connecting event mentions—followed by the set of marked mentions that satisfy the relation. The bipartite matching loss uses the Hungarian algorithm to align the generated answer list with the gold list under the best permutation, so the order of the answers has no effect on the gradient; the dependency parsing chain is trained with ordinary cross-entropy and provides the structured scaffold that makes scattered event relations recoverable.","core_discovery":"On its own terms, the paper's discovery is that the bottleneck in instruction-based event relation extraction is the question format, not the language model. Instead of enumerating all event pairs and asking 'what relation holds?', MAQInstruct asks 'which marked events stand in relation R to the query event?', with the context carrying the candidate mentions as special marker tokens. The label is a dependency parsing chain followed by a comma-separated list of answers; the loss over the answer list is computed with a Hungarian matching, so any permutation of the generated answers receives the same loss. Experiments show this formulation improves F1 over InstructERE by roughly four points per subtask and over the classification-based BertERE by 0.4 to 1.7 points on MAVEN-ERE, and it transfers zero-shot to HiEve, MATRES, and MECI better than the prior instruction method.","pith_inferences":["The relation-as-instruction, mentions-as-answer inversion is a general template: any task with few relation or role types and many candidate spans (for example argument role labeling, entity linking, or coreference beyond events) could inherit the same sample-count reduction.","Because the dependency parsing chain is itself a generated sequence in a fixed order, the paper's sequence-independence claim is only partially demonstrated; a stress test would shuffle the chain's edges at inference, and if accuracy drops, the ordering problem has moved to the chain rather than disappearing.","The efficiency advantage depends on $k \\ll n$; on tasks with a large relation inventory, the gap between $k \\times n$ and $n^2$ closes, so the method is most compelling in sparse-relation regimes.","The ablation shows markers alone carry most of the gain when the chain is present, suggesting that marker-anchored decoding, rather than the chain's linguistic content per se, might be the active ingredient; a test would be to replace the dependency parse with a random but consistent structured scaffold."],"forward_implications":["A single instruction-tuned LLM can extract coreference, temporal, causal, and sub-event relations using the same framework, with only the instruction changing.","The number of inference samples scales with the number of relation types times the number of mentions rather than the square of the mention count, making document-level extraction with dozens or hundreds of mentions feasible for LLMs.","The bipartite matching loss makes answer generation order irrelevant, which the paper shows holds across random, reverse, distance-sorted, and dictionary-sorted answer orders.","The dependency parsing chain is the largest accuracy contributor: removing it drops MATRES F1 from 84.7 to 81.4 and MECI F1 from 62.3 to 58.4, so structured syntactic scaffolding is central to the gain.","Zero-shot performance on unseen event-relation datasets (HiEve, MATRES, MECI) improves relative to the prior instruction-based method, and general NLU performance is not degraded."],"supporting_citations":[{"why":"Supplies MAVEN-ERE, the unified event-relation benchmark with coreference, temporal, causal, and sub-event subtasks; this is the paper's main evaluation set.","marker":"[31]"},{"why":"MATRES is the temporal relation dataset used for the order-sensitivity experiments and for model ablations.","marker":"[21]"},{"why":"MECI is the causal relation dataset used for the order-sensitivity experiments and for model ablations.","marker":"[14]"},{"why":"HiEve is the sub-event relation dataset used for zero-shot transfer evaluation.","marker":"[8]"},{"why":"Presents the sequence-to-set optimal transport loss that motivates the use of a bipartite matching loss for unordered answers.","marker":"[3]"},{"why":"Argues that generated sequence order affects text generation quality, providing the rationale for making the answer order irrelevant.","marker":"[40]"},{"why":"Provides the unified structure generation approach whose marker-based output format MAQInstruct builds on.","marker":"[16]"},{"why":"Introduces the packed levitated marker technique that inspires the <0x64>-<0xFF> mention markers used in the instructions.","marker":"[39]"}],"fun_headline_variants":["Event-relation extraction: flip the question to select events, not relations","MAQInstruct: ask 'which events fit this relation?' instead of 'what relation?'","Order-free event-relation extraction: Hungarian matching beats sequence loss","Event-relation extraction: fewer inference queries, higher F1 across LLMs","MAQInstruct: invert event-relation instructions to cut queries and boost F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The premise that carries the method is that the model can generate the dependency parsing chain in its fixed order, and that this chain is what makes the scattered event relations recoverable; the ablation shows that without the chain the method loses the most accuracy, so if chain order is itself unreliable the claimed sequence independence is only shifted, not eliminated.","fun_headline_variants_meta":{"raw":{"variants":["Event-relation extraction: flip the question to select events, not relations","MAQInstruct: ask 'which events fit this relation?' instead of 'what relation?'","Order-free event-relation extraction: Hungarian matching beats sequence loss","Event-relation extraction: fewer inference queries, higher F1 across LLMs","MAQInstruct: invert event-relation instructions to cut queries and boost F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1750,"prompt_tokens":875,"completion_tokens":875,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":772}},"tokens_in":491,"tokens_out":875,"duration_ms":8957,"temperature":1.0,"reasoning_tokens":772,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T00:06:15.053230+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the order of the edges in the dependency parsing chain at inference time (for example, reverse the chain, or sort edges randomly) while keeping the same answers; if MATRES or MECI F1 drops materially below the reported numbers, the method still depends on generation order, contradicting the claim that the bipartite matching loss removes sequence sensitivity. Alternatively, replace the generated chain with an unordered set of dependency edges; if accuracy collapses, the chain is functioning as a fixed-order crutch rather than a content scaffold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MAVEN-ERE, the unified event-relation benchmark with coreference, temporal, causal, and sub-event subtasks; this is the paper's main evaluation set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MATRES is the temporal relation dataset used for the order-sensitivity experiments and for model ablations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MECI is the causal relation dataset used for the order-sensitivity experiments and for model ablations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HiEve is the sub-event relation dataset used for zero-shot transfer evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents the sequence-to-set optimal transport loss that motivates the use of a bipartite matching loss for unordered answers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Argues that generated sequence order affects text generation quality, providing the rationale for making the answer order irrelevant."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the unified structure generation approach whose marker-based output format MAQInstruct builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the packed levitated marker technique that inspires the <0x64>-<0xFF> mention markers used in the instructions."}],"review_version":1}