REVIEW 3 major objections 5 minor 1 cited by
MAQInstruct: Instruction-based Unified Event Relation Extraction
T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read MAQInstruct recasts event-relation extraction as multiple-answer selection among marked event mentions, using a bipartite matching loss to remove answer-order effects and cutting inference queries from $n^2$ to $k \times n$.
desk verdict Solid efficiency win for instruction-based ERE, but the order-invariance claim only covers the answer list, not the dependency chain that actually drives performance. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the multiple-answer question format plus a bipartite matching loss. Each event mention in the context is preceded by a distinct marker token (for example <0x85>), and an instruction names one relation and one query mention; the model must output a dependency parsing chain—a sequence of typed syntactic edges connecting event mentions—followed by the set of marked mentions that satisfy the relation. The bipartite matching loss uses the Hungarian algorithm to align the generated answer list with the gold list under the best permutation, so the order of the answers has no effect on the gradient; the dependency parsing chain is trained with ordinary cross-entropy and provides the structured scaffold that makes scattered event relations recoverable.
What would settle it
Shuffle the order of the edges in the dependency parsing chain at inference time (for example, reverse the chain, or sort edges randomly) while keeping the same answers; if MATRES or MECI F1 drops materially below the reported numbers, the method still depends on generation order, contradicting the claim that the bipartite matching loss removes sequence sensitivity. Alternatively, replace the generated chain with an unordered set of dependency edges; if accuracy collapses, the chain is functioning as a fixed-order crutch rather than a content scaffold.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that the bottleneck in instruction-based event relation extraction is the question format, not the language model. Instead of enumerating all event pairs and asking 'what relation holds?', MAQInstruct asks 'which marked events stand in relation R to the query event?', with the context carrying the candidate mentions as special marker tokens. The label is a dependency parsing chain followed by a comma-separated list of answers; the loss over the answer list is computed with a Hungarian matching, so any permutation of the generated answers receives the same loss. Experiments show this formulation improves F1 over InstructERE by roughly four points per subtask and over the classification-based BertERE by 0.4 to 1.7 points on MAVEN-ERE, and it transfers zero-shot to HiEve, MATRES, and MECI better than the prior instruction method.
Load-bearing premise
The premise that carries the method is that the model can generate the dependency parsing chain in its fixed order, and that this chain is what makes the scattered event relations recoverable; the ablation shows that without the chain the method loses the most accuracy, so if chain order is itself unreliable the claimed sequence independence is only shifted, not eliminated.
Editorial extensions
If this is right
- A single instruction-tuned LLM can extract coreference, temporal, causal, and sub-event relations using the same framework, with only the instruction changing.
- The number of inference samples scales with the number of relation types times the number of mentions rather than the square of the mention count, making document-level extraction with dozens or hundreds of mentions feasible for LLMs.
- The bipartite matching loss makes answer generation order irrelevant, which the paper shows holds across random, reverse, distance-sorted, and dictionary-sorted answer orders.
- The dependency parsing chain is the largest accuracy contributor: removing it drops MATRES F1 from 84.7 to 81.4 and MECI F1 from 62.3 to 58.4, so structured syntactic scaffolding is central to the gain.
- Zero-shot performance on unseen event-relation datasets (HiEve, MATRES, MECI) improves relative to the prior instruction-based method, and general NLU performance is not degraded.
Reading between the lines
- The relation-as-instruction, mentions-as-answer inversion is a general template: any task with few relation or role types and many candidate spans (for example argument role labeling, entity linking, or coreference beyond events) could inherit the same sample-count reduction.
- Because the dependency parsing chain is itself a generated sequence in a fixed order, the paper's sequence-independence claim is only partially demonstrated; a stress test would shuffle the chain's edges at inference, and if accuracy drops, the ordering problem has moved to the chain rather than disappearing.
- The efficiency advantage depends on $k \ll n$; on tasks with a large relation inventory, the gap between $k \times n$ and $n^2$ closes, so the method is most compelling in sparse-relation regimes.
- The ablation shows markers alone carry most of the gain when the chain is present, suggesting that marker-anchored decoding, rather than the chain's linguistic content per se, might be the active ingredient; a test would be to replace the dependency parse with a random but consistent structured scaffold.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MAQInstruct, an instruction-tuned framework for unified event relation extraction. It replaces pairwise event-event questions with relation-specific instructions that ask the model to select one or more event mentions from the context, reducing the number of inference queries from quadratic to linear in the number of mentions per document. It also introduces a bipartite matching loss (BPM) applied to the multiple-answer portion of the generated output, intended to make the model insensitive to the order in which answers are generated. Experiments on MAVEN-ERE compare MAQInstruct with BertERE and an InstructERE baseline across three LLMs (ChatGLM3, Qwen, Llama2), and cross-corpus/zero-shot results are reported on HiEve, MATRES, and MECI. The paper reports consistent F1 improvements over InstructERE (about 3.9-4.2 points on the four MAVEN-ERE subtasks for Llama2) and an inference-time reduction from 813 to 25 minutes for coreference on an A100-80G setup.
Significance. The empirical scope is a genuine strength: the method is evaluated on four ERE datasets, three LLM backbones, and five seeds, and the inference-cost comparison in Table 2 is concrete and central. The main efficiency claim (quadratic to linear query count) is well motivated and supported by the reported costs. If the order-invariance claim were fully established, the BPM component would be a useful and transferable idea. However, the present evidence supports order-invariance only for the final answer list, not for the full generated output, which weakens the contribution's stated scope.
major comments (3)
- [§2.2, Eq. (4); §3.6, Table 3] The claim that MAQInstruct reduces the dependency on the generation sequence is only partially supported. The BPM loss in Eq. (4) is applied only to the multiple-answer part; the dependency parsing chain (DPC) is generated with ordinary cross-entropy under a fixed order (Appendix A), and Table 3 shows that removing the DPC causes the largest performance drop (MATRES F1 84.7 to 81.4; MECI F1 62.3 to 58.4). No experiment varies the DPC order, although Appendix E itself attributes errors to 'the complexity of the dependency parsing chain.' The paper should either restrict the order-invariance claim to the multiple-answer list or demonstrate that the DPC is order-stable (e.g., by comparing parser-determined, shuffled, and reversed DPC orders).
- [Table 1; Appendix B] The comparison against BertERE is presented as a positive result, but the gains are as small as 0.4 and at most 1.7 F1 points, and no standard deviations or significance tests are reported even though Appendix B states the results are averaged over five random seeds. Without error bars or a paired test, the claim that MAQInstruct 'enhances' over BertERE is not statistically supported; at minimum, report per-seed variance or a significance test for the four MAVEN-ERE subtasks.
- [§2.2, Eqs. (2)-(3)] The formal definition of the matching loss is ambiguous. N2 is introduced as the length of the multiple-answer part in tokens, but the permutation Ψ_N2 in Eqs. (2)-(3) is over token positions, with c_i a target vocabulary id; for multi-token answers, matching individual tokens is not equivalent to matching whole answer spans. Please state clearly whether the Hungarian algorithm operates over answer spans or tokens, and how variable-length answers are handled; this is necessary to reproduce the loss exactly.
minor comments (5)
- [Section 3.1] InstructERE is never cited or referenced; please either cite the original work or state explicitly that it is an internal baseline, and if it is based on the authors' prior ChatUIE [37], say so.
- [Section 3.2] The claimed '4.9%' improvement for coreference over InstructERE does not match any of the four coreference metrics in Table 1 (the closest is 4.2 points on BLANC); please correct the number.
- [Figure 3] The plot has no axis labels or legend entries explaining what is being measured; please add labels for F1 and for the dataset/sequence-condition grouping.
- [Section 3.5] The term 'zero-shot' is used for models trained on MAVEN-ERE and evaluated on HiEve/MATRES/MECI; this is cross-corpus transfer rather than zero-shot learning in the usual sense, so the terminology should be clarified.
- [Table 2; Section 1] The n×n sample count should specify that n is the number of event mentions per document rather than the total number of mentions in the dataset; otherwise the reported 631,486 coreference queries for InstructERE appear inconsistent with the 17,780 total mentions.
Circularity Check
No circularity: the reported gains are measured on external benchmarks and the bipartite matching loss is an externally established set-generation technique applied to ERE, not a fitted input renamed as a prediction.
full rationale
MAQInstruct's central claims are empirical: F1 improvements on MAVEN-ERE, HiEve, MATRES, and MECI, an inference-time reduction from explicit query counts, and ablation comparisons in Table 3. None of these numbers is obtained by fitting a parameter to a quantity and then reporting that same quantity as a prediction. The k×n versus n×n sample reduction is an arithmetic consequence of the task reformulation rather than a discovered result, and the paper does not disguise it as a prediction. The bipartite matching loss is imported from the externally cited set-generation literature (OTSeq2Set [3], One2Set [40]), where permutation-invariance is a known property of the Hungarian matching objective; Section 3.4 tests the loss under different answer orderings rather than assuming its effect. The only self-citations ([37], [38]) appear in standard metric and related-work reference lists and are not load-bearing for the paper's mechanism. The dependency-parsing-chain order dependence raised in the skeptical reading is a genuine limitation: the DPC is an ordered generated sequence, and its removal causes the largest ablation drop, so the 'generation sequence' claim strictly covers the multiple-answer block rather than the entire output. But that is a scope/generality gap, not a circular reduction: the DPC is produced by the external CoreNLP parser with deterministic tie-breaking and is evaluated by ablation, so the derivation chain does not assume what it purports to prove. The InstructERE baseline is described without a citation, which is a source-attribution concern, but it is not an example of a result reducing to its own input.
Assumptions & free parameters
free parameters (1)
- lambda (bipartite matching loss weight) =
0.2
assumptions (4)
- standard math The Hungarian algorithm finds the optimal permutation for token-level answer alignment.
- domain assumption The dependency parser (CoreNLP) produces dependency chains that are valid and useful for ERE.
- domain assumption Instruction-tuned LLMs can map relation-specific instructions to correct event selections.
- domain assumption The multiple-answer QA formulation can recover all relations from k*n queries without needing pair-wise information.
Cite this review
Pith. "Pith review of MAQInstruct: Instruction-based Unified Event Relation Extraction." pith.science (2026). https://pith.science/paper/K6SX36NR
@misc{pith2026250203954,
author = {Pith},
title = {Pith review of: MAQInstruct: Instruction-based Unified Event Relation Extraction},
year = {2026},
howpublished = {\url{https://pith.science/paper/K6SX36NR}},
note = {Machine review of arXiv:2502.03954}
}
read the original abstract
Extracting event relations that deviate from known schemas has proven challenging for previous methods based on multi-class classification, MASK prediction, or prototype matching. Recent advancements in large language models have shown impressive performance through instruction tuning. Nevertheless, in the task of event relation extraction, instruction-based methods face several challenges: there are a vast number of inference samples, and the relations between events are non-sequential. To tackle these challenges, we present an improved instruction-based event relation extraction framework named MAQInstruct. Firstly, we transform the task from extracting event relations using given event-event instructions to selecting events using given event-relation instructions, which reduces the number of samples required for inference. Then, by incorporating a bipartite matching loss, we reduce the dependency of the instruction-based method on the generation sequence. Our experimental results demonstrate that MAQInstruct significantly improves the performance of event relation extraction across multiple LLMs.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
GDLLM: A Global Distance-aware Modeling Approach Based on Large Language Models for Event Temporal Relation Extraction
GDLLM improves event temporal relation extraction by feeding LLM-generated probability distributions into a graph attention network, achieving state-of-the-art micro-F1 scores on TB-Dense and MATRES.
Reference graph
Works this paper leans on
-
[1]
Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The first international conference on language resources and evaluation workshop on linguistics coreference, Vol. 1. 563–566
work page 1998
-
[2]
Shany Barhom, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019. Revisiting Joint Modeling of Cross-document Entity and Event Coreference Resolution. In ACL. 4179–4189
work page 2019
-
[3]
Jie Cao and Yin Zhang. 2022. OTSeq2Set: An Optimal Transport Enhanced Sequence-to-Set Model for Extreme Multi-label Text Classification. In EMNLP. 5588–5597
work page 2022
-
[4]
Tommaso Caselli and Piek Vossen. 2017. The Event StoryLine Corpus: A New Benchmark for Causal and Temporal Relation Extraction. In ACL. 77–86
work page 2017
-
[5]
Meiqi Chen, Yubo Ma, Kaitao Song, Yixin Cao, Yan Zhang, and Dongsheng Li
-
[6]
Prafulla Kumar Choubey and Ruihong Huang. 2017. Event Coreference Reso- lution by Iteratively Unfolding Inter-dependencies among Events. In EMNLP. 2124–2133
work page 2017
-
[7]
Shiyao Cui, Jiawei Sheng, Xin Cong, Quangang Li, Tingwen Liu, and Jinqiao Shi
-
[8]
Goran Glavas, Jan Snajder, Marie-Francine Moens, and Parisa Kordjamshidi. 2014. HiEve: A Corpus for Extracting Event Hierarchies from News Stories. In LREC. 3678–3683
work page 2014
Show all 46 references
-
[9]
Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. 2019. A Multi- Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning. In EMNLP-IJCNLP. Hong Kong, China
2019
-
[10]
Zhilei Hu, Zixuan Li, Xiaolong Jin, Long Bai, Saiping Guan, Jiafeng Guo, and Xueqi Cheng. 2023. Semantic Structure Enhanced Event Causality Identification. In ACL. 10901–10913
2023
-
[11]
Zhilei Hu, Zixuan Li, Daozhu Xu, Long Bai, Cheng Jin, Xiaolong Jin, Jiafeng Guo, and Xueqi Cheng. 2023. ProtoEM: A Prototype-Enhanced Matching Framework for Event Relation Extraction. arXiv:2309.12892 [cs.CL]
2023 arXiv
-
[12]
Quzhe Huang, Yutong Hu, Shengqi Zhu, Yansong Feng, Chang Liu, and Dongyan Zhao. 2023. More than Classification: A Unified Framework for Event Temporal Relation Extraction. In ACL. 9631–9646
2023
-
[13]
EunJeong Hwang, Jay-Yoon Lee, Tianyi Yang, Dhruvesh Patel, Dongxu Zhang, and Andrew McCallum. 2022. Event-Event Relation Extraction using Probabilistic Box Embedding. In ACL. 235–244
2022
-
[14]
Viet Dac Lai, Amir Pouran Ben Veyseh, Minh Van Nguyen, Franck Dernoncourt, and Thien Huu Nguyen. 2022. MECI: A Multilingual Dataset for Event Causality Identification. In COLING. 2346–2356
2022
-
[15]
Jing Lu and Vincent Ng. 2021. Constrained Multi-Task Learning for Event Coreference Resolution. In NAACL-HLT. 4504–4514. MAQInstruct: Instruction-based Unified Event Relation Extraction WWW Companion ’25, April 28-May 2, 2025, Sydney, NSW, Australia
2021
-
[16]
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Unified Structure Generation for Universal Information Extraction. In ACL. Dublin, Ireland, 5755–5772
2022
-
[17]
Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In HLT (Vancouver, British Columbia, Canada). Morristown, NJ, USA, 25–32
2005
-
[18]
Hieu Man, Nghia Trung Ngo, Linh Ngo Van, and Thien Huu Nguyen. 2022. Selecting Optimal Context Sentences for Event-Event Relation Extraction. In AAAI. 11058–11066
2022
-
[19]
Minh Van Nguyen, Bonan Min, Franck Dernoncourt, and Thien Nguyen. 2022. Learning Cross-Task Dependencies for Joint Extraction of Entities, Events, Event Arguments, and Relations. In EMNLP. 9349–9360
2022
-
[20]
Minh Van Nguyen, Bonan Min, Franck Dernoncourt, and Thien Huu Nguyen
-
[21]
Qiang Ning, Hao Wu, and Dan Roth. 2018. A Multi-Axis Annotation Scheme for Event Temporal Relations. In ACL. 1318–1328
2018
-
[22]
RECASENS and E
M. RECASENS and E. HOVY. 2011. BLANC: Implementing the Rand index for coreference evaluation. Natural Language Engineering 17, 4 (2011), 485–510
2011
-
[23]
In NAACL
Joint Extraction of Entities, Relations, and Events via Modeling Inter- Instance and Inter-Label Dependencies. In NAACL. 4363–4374
-
[24]
Shirong Shen, Heng Zhou, Tongtong Wu, and Guilin Qi. 2022. Event Causality Identification via Derivative Prompt Joint Learning. In COLING. 2288–2299
2022
-
[25]
Xingwei Tan, Gabriele Pergola, and Yulan He. 2023. Event Temporal Relation Extraction with Bayesian Translational Model. In EACL. 1117–1130
2023
-
[26]
Elad Segal, Avia Efrat, Mor Shoham, Amir Globerson, and Jonathan Berant. 2020. A Simple and Effective Model for Answering Multi-span Questions. In EMNLP. 3074–3080
2020
-
[27]
Vilain, John D
Marc B. Vilain, John D. Burger, John S. Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A Model-Theoretic Coreference Scoring Scheme. In Message Understanding Conference
1995
-
[28]
Somin Wadhwa, Silvio Amir, and Byron C. Wallace. 2023. Revisiting Relation Extraction in the era of Large Language Models. In ACL. 15566–15589
2023
-
[29]
Hieu Minh Tran, Duy Phung, and Thien Huu Nguyen. 2021. Exploiting Docu- ment Structures and Cluster Consistencies for Event Coreference Resolution. In ACL/IJCNLP. 4840–4850
2021
-
[30]
Gardner, Dan Roth, and Muhao Chen
Haoyu Wang, Hongming Zhang, Yuqian Deng, Jacob R. Gardner, Dan Roth, and Muhao Chen. 2023. Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extraction. In EACL. 541–553
2023
-
[31]
Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie Zhou. 2022. MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction. In EMNLP. 926–941
2022
-
[32]
Haoyu Wang, Muhao Chen, Hongming Zhang, and Dan Roth. 2020. Joint Con- strained Learning for Event-Event Relation Extraction. In EMNLP. 696–706
2020
-
[33]
Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jihua Kang, Jingsheng Yang, Siyuan Li, and Chunsai Du. 2023. InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction. arXiv:2304.08085 [cs.CL]
2023 arXiv
-
[34]
Haoyang Wen and Heng Ji. 2021. Utilizing Relative Event Time to Enhance Event-Event Temporal Relation Extraction. In EMNLP. 10431–10437
2021
-
[35]
Xiaozhi Wang, Hao Peng, Yong Guan, Kaisheng Zeng, Jianhui Chen, Lei Hou, Xu Han, Yankai Lin, Zhiyuan Liu, Ruobing Xie, Jie Zhou, and Juanzi Li. 2024. MAVEN-ARG: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation. In ACL. 4072–4091
2024
-
[36]
Xinglin Xiao, Yijie Wang, Nan Xu, Yuqi Wang, Hanxuan Yang, Minzheng Wang, Yin Luo, Lei Wang, Wenji Mao, and Daniel Zeng. 2024. YAYI-UIE: A Chat- Enhanced Instruction Tuning Framework for Universal Information Extraction. arXiv:2312.15548 [cs.CL]
2024 arXiv
-
[37]
Jun Xu, Mengshu Sun, Zhiqiang Zhang, and Jun Zhou. 2024. ChatUIE: Exploring Chat-based Unified Information Extraction Using Large Language Models. In LREC/COLING 2024. 3146–3152
2024
-
[38]
Wei Xiang, Chuanhong Zhan, and Bang Wang. 2023. DAPrompt: Determin- istic Assumption Prompt Learning for Event Causality Identification. CoRR abs/2307.09813 (2023)
2023 arXiv
-
[39]
Deming Ye, Yankai Lin, Peng Li, and Maosong Sun. 2022. Packed Levitated Marker for Entity and Relation Extraction. In ACL. 4904–4917
2022
-
[40]
Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. 2021. One2Set: Gener- ating Diverse Keyphrases as a Set. In ACL/IJCNLP. 4598–4608
2021
-
[41]
Jun Xu, Weidi Xu, Mengshu Sun, Taifeng Wang, and Wei Chu. 2022. Extracting Trigger-sharing Events via an Event Matrix. In EMNLP. Abu Dhabi, United Arab Emirates, 1189–1201
2022
-
[42]
Yutao Zeng, Xiaolong Jin, Saiping Guan, Jiafeng Guo, and Xueqi Cheng. 2020. Event Coreference Resolution with their Paraphrases and Argument-aware Em- beddings. In COLING. 3084–3094
2020
-
[43]
" with
Jie Zhou, Shenpo Dong, Hongkui Tu, Xiaodong Wang, and Yong Dou. 2022. RSGT: Relational Structure Guided Temporal Relation Extraction. In COLING. 2001–2010. A Dependency Parsing Chain We use the Stanford NLP toolkit’s CoreNLP Dependency Parser to create a dependency parse tree ...
2022
-
[44]
Changsen Yuan, Heyan Huang, Yixin Cao, and Yonggang Wen. 2023. Discrimi- native Reasoning with Sparse Event Representation for Document-level Event- Event Relation Extraction. In ACL. 16222–16234
2023
-
[2022]
In COLING
Event Causality Extraction with Event Argument Correlations. In COLING. 2300–2312
-
[2024]
Improving Large Language Models in Event Relation Logical Prediction. In ACL. 9451–9478
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.