REVIEW 3 major objections 4 minor 40 references
Keyword-Centric Prompting for One-Shot Event Detection with Self-Generated Rationale Enhancements
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that adding a small set of trigger keywords plus automatically generated propose-and-judge rationales to the prompt lifts one-shot event detection above both previous in-context learning and supervised fine-tuning, reaching
desk verdict A solid, well-ablated prompt-engineering recipe for one-shot event detection, but the headline margin is likely inflated by test-set tuning of S and an omitted strong baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the two-stage prompt: (1) keyword anchoring, where a small set of exemplary triggers (e.g., 'lend', 'borrow', 'give' for Transaction.Transfer-Money) is injected into the event description and matched against the input text via lemmatized string matching; (2) proposal-judgment rationales, where the model first proposes trigger candidates that are not in the keyword set, then writes a sentence explaining why each candidate does or does not satisfy the event definition. The rationales for the in-context demonstrations are produced automatically by the same LLM in a three-step pipeline (candidate probing, negative sampling with probability proportional to $e^{|C_t|/
What would settle it
Run KeyCP++ on a new event ontology with abstract, lexically unmarked event types (e.g., Cognitive.Identify or Justice.Appeal) and on a non-English corpus where the GPT3.5-generated keyword list is visibly low quality; if F1 falls below the vanilla-prompting baseline or below the DEGREE supervised baseline on the same split, the keyword-anchor mechanism is not portable. The paper's own Appendix B admits low-quality keyword generations, so a controlled study with hand-curated versus generated keywords for the same event types would settle how much of the gain depends on keyword quality.
Extended reading notes
Core claim
KeyCP++ is a chain-of-thought prompting framework for event detection in a one-shot setting. It first gives the model a list of keywords, words that typically trigger the target event type, and uses string matching to tell the model which of them appear in the query. Then, instead of stopping there, it instructs the model to propose additional candidate trigger words beyond the keyword list and to write a judgment for each candidate: is this word actually consistent with the event definition in this context? The model learns this propose-and-judge pattern from demonstrations whose rationales are generated automatically by the same kind of model, using candidate probing, negative sampling bia
Load-bearing premise
The whole method leans on the set of keywords generated by GPT3.5 from event definitions being a sufficient and stable anchor for trigger profiling across domains, models, and languages; if those keywords are often wrong or incomplete, the keyword-anchored prompting collapses (removing them drops F1 from 45.1 to 24.2).
Editorial extensions
If this is right
- If the method holds, one-shot event detection no longer requires supervised fine-tuning for a new event type; a definition, a few keywords, and one labeled example per type are enough to exceed the best previous ICL and SFT results.
- The automatic rationale generation removes the human-annotation bottleneck for chain-of-thought demonstrations, making the approach scalable to continuously emerging event types.
- The ablation shows that the judgment step is the most critical component: removing it drops F1 from 45.1 to 34.8 on ACE05-E with LLaMA2-13B, so the rationale's discrimination role, not just the keywords, carries the gain.
- The method transfers to a 2–4 shot regime, and gains widen as more shots become available, suggesting the proposal-judgment format continues to extract useful signal from additional negative examples.
- Performance varies by event type, with some types (e.g., Justice.Appeal, Justice.Pardon) still difficult; the method does not eliminate all trigger ambiguity, as the Life.Marry/divorce case shows.
Reading between the lines
- A reader could infer that the propose-and-judge loop extends to event argument extraction or relation extraction, where the same over-interpretation problem appears; the machinery is task-agnostic as long as a definition and an anchor set can be written.
- The method's reliance on GPT3.5-generated keywords is a clear pressure point: the appendix admits many low-quality generations, so testing with hand-curated or automatically learned keywords for non-English text is a natural next experiment.
- The negative-sampling bias toward examples with many candidates is a distinctive design choice; one could test whether an even more extreme bias (higher temperature) or a diversity penalty over candidate types improves discrimination between confusable event types like Conflict.Attack and Life.Injure.
- If the gains persist across models of different sizes and families, the approach suggests that for structured extraction tasks, the format of the prompt matters more than the choice of LLM, a claim worth checking against larger and newer models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes KeyCP and KeyCP++ for one-shot event detection. KeyCP augments vanilla ICL prompts with event-type keywords (automatically generated by GPT-3.5 from definitions) plus string-matching keyword detection at inference; KeyCP++ additionally inserts automatically generated 'propose-and-judge' rationales into the demonstrations, where candidate triggers are probed and then accepted or rejected against the event definition. The method is evaluated on ACE05-E, ACE05-E+, and WikiEvents with LLaMA2-13B, Mistral-7B, GPT-3.5, and DeepSeek-V3. The central empirical claim is that KeyCP++ substantially outperforms prior ICL methods and supervised fine-tuning baselines, e.g., 57.6 F1 on ACE05-E with DeepSeek-V3 versus 45.1 for CodeUIE and 44.8 for DEGREE. Ablations show that each component (keywords, negative sampling, probing, proposals, judgment) contributes to the final performance.
Significance. If the reported gains hold, this is a practically useful contribution: it removes the need for human-written rationale annotations in chain-of-thought prompting for event detection, and it shows consistent improvements across four LLMs of different families and scales. The paper ships fairly thorough ablations (Table 4), per-event-type analysis (Appendix A), and case studies, which support the claim that the components are doing real work. The automatic rationale generation via candidate probing and judgment is a sensible recipe. The main caveats are methodological: the number of negative examples S appears to be selected on the evaluation set, and the strongest baseline comparisons are reproduced on a different model than the one originally reported.
major comments (3)
- [Section 5.4, Figure 6, Eq. (1)] The headline results in Tables 1–2 all use S=5 negative examples, but Section 5.4 states that 'the best performance occurs at S=5' after sweeping S on ACE-05. No held-out development split is described for this selection. If S=5 was chosen by inspecting test-set performance, the reported F1 values are optimistically biased. Please report the selection procedure explicitly, or re-run the headline comparisons with S fixed a priori or chosen on a true validation split, and show the sensitivity across S.
- [Section 4.2, Table 2] The ChatGPT and CodeUIE baselines in Table 2 are reproduced using DeepSeek-V3 rather than the original models (GPT-3.5 and GPT-3.5). While this makes the comparison model-matched for KeyCP++, the table labels are misleading, and the comparison is not with the originally reported numbers. The paper also evaluates GPT-3.5, so it should report original GPT-3.5-based baseline numbers or additional reproductions on GPT-3.5. Please add a column or footnote stating which model produced each baseline row.
- [Section 2.2, Table 2] Guideline Learning (Pang et al., 2023) is described as the closest prior ICL approach that injects automatically generated guidelines into the prompt, yet it is omitted from Table 2. The stated reason—that it handles classification but not identification—is an adaptation issue rather than a demonstration that it cannot serve as a strong baseline. Please include an adapted Guideline Learning baseline or explicitly justify with experiments why it is not comparable for trigger identification. As written, the 'previous in-context learning methods' comparison is incomplete.
minor comments (4)
- [Section 5.4 / Appendix B] The table numbering is inconsistent: Section 5.4 refers to 'Table 6' for negative-example sensitivity, while Appendix B also labels the keyword-generation prompt as 'Table 6'. This should be corrected.
- [Appendix B] The sentence 'Although we have leverage voting in the candidate generation' contains a grammatical error ('leverage' should be 'leveraged' or 'we use voting'). Also, the appendix admits 'there are still many low-quality generations'; this should be discussed more directly in the main text given the strong dependence on keyword quality shown in Table 4.
- [Abstract / Introduction] The claim 'we are the first to present an effective chain-of-thought prompting paradigm for event detection' is too strong and is not supported by a systematic literature review. I recommend softening it to 'the first, to our knowledge' and citing related CoT-for-event-extraction work if any exists.
- [Table 2] Each baseline row should explicitly state the underlying LLM, e.g., 'ChatGPT (reproduced with DeepSeek-V3)', to prevent readers from assuming the original model was used.
Circularity Check
Headline F1 is partially selected: S=5 is chosen on the same test set used to report the 57.6 F1 result, though the prompting method itself is not derived from its outputs.
-
fitted input called prediction
[Section 3.2 Eq. (1); Section 5.4 Fig. 6/Table 6]
"In our experiments, we set the negative sampling sizeS = 5. ... Table 6 shows the performance of KeyCP++ for varying numbers of negative examples. We find the best performance occurs atS = 5."
The hyperparameter S (number of negative demonstrations) is selected by sweeping on the ACE05-E test set and taking the best F1 point (Figure 6, Section 5.4). The same configuration is then used to produce the headline results in Tables 1 and 2, including 57.6 F1 on ACE05-E. The reported number is therefore the maximum of the test-set sweep rather than an independent evaluation: the test set has been used to choose the prompt, so the headline comparison is partially constructed by the selection procedure rather than predicted.
full rationale
The paper is an empirical prompting study with no formal derivation chain, so the main circularity failure modes (self-definitional equations, imported uniqueness theorems, ansatz smuggling) are absent. The central construction--keyword anchoring plus automatic propose-and-judge rationales--is not fitted to the evaluation labels; the rationales are generated from one-shot training examples and event definitions, and the comparison against Vanilla/KeyCP and prior ICL/SFT baselines provides independent content. The one concrete circularity-adjacent issue is the test-set selection of S=5: the paper explicitly chooses S by maximizing F1 on ACE05-E and then reports results with that S on the same dataset, which inflates the headline numbers and makes the ACE05-E comparison partially a selection artifact. This affects the strength of the central quantitative claim but does not reduce the method's logic to its inputs. The exclusion of Guideline Learning from Table 2 and the acknowledged low-quality keyword generations are correctness/robustness concerns, not circularity.
Assumptions & free parameters
free parameters (4)
- Number of negative examples S =
5
- Negative sampling temperature tau =
1
- Candidate probing voting threshold =
>3 out of 5
- Rationale generation temperature / top-p =
0.9 / 0.6
assumptions (4)
- domain assumption Event definitions from ACE2005 annotation guidelines and KAIROS ontology are accurate and sufficient for the task
- domain assumption GPT3.5-generated keywords are stable anchors for trigger profiling
- domain assumption LLM-generated rationales serve as valid demonstrations without human verification
- domain assumption The random one-shot training splits are representative of the task
Cite this review
Pith. "Pith review of Keyword-Centric Prompting for One-Shot Event Detection with Self-Generated Rationale Enhancements." pith.science (2026). https://pith.science/paper/HZKYSTAG
@misc{pith2026250807598,
author = {Pith},
title = {Pith review of: Keyword-Centric Prompting for One-Shot Event Detection with Self-Generated Rationale Enhancements},
year = {2026},
howpublished = {\url{https://pith.science/paper/HZKYSTAG}},
note = {Machine review of arXiv:2508.07598}
}
read the original abstract
Although the LLM-based in-context learning (ICL) paradigm has demonstrated considerable success across various natural language processing tasks, it encounters challenges in event detection. This is because LLMs lack an accurate understanding of event triggers and tend to make over-interpretation, which cannot be effectively corrected through in-context examples alone. In this paper, we focus on the most challenging one-shot setting and propose KeyCP++, a keyword-centric chain-of-thought prompting approach. KeyCP++ addresses the weaknesses of conventional ICL by automatically annotating the logical gaps between input text and detection results for the demonstrations. Specifically, to generate in-depth and meaningful rationale, KeyCP++ constructs a trigger discrimination prompting template. It incorporates the exemplary triggers (a.k.a keywords) into the prompt as the anchor to simply trigger profiling, let LLM propose candidate triggers, and justify each candidate. These propose-and-judge rationales help LLMs mitigate over-reliance on the keywords and promote detection rule learning. Extensive experiments demonstrate the effectiveness of our approach, showcasing significant advancements in one-shot event detection.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
S. Bird. NLTK: The Natural Language Toolkit. In J. Curran, editor,Pro- ceedings of the COLING/ACL 2006 Interactive Presentation Sessions , Sydney, Australia, July 2006. Association for Computational Linguis- tics
work page 2006
-
[2]
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amode...
work page 2020
-
[3]
R. Chen, C. Qin, W. Jiang, and D. Choi. Is a large language model a good annotator for event extraction? In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 38, pages 17772–17780, 2024
work page 2024
-
[4]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025
DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025
2025
-
[5]
G. R. Doddington, A. Mitchell, M. A. Przybocki, L. A. Ramshaw, S. M. Strassel, and R. M. Weischedel. The automatic content extrac- tion (ACE) program - tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evalua- tion, LREC 2004, May 26-28, 2004, Lisbon, Portugal . European Lan- guage Resources Assoc...
work page 2004
- [6]
-
[7]
J. Gao, H. Zhao, C. Yu, and R. Xu. Exploring the Feasibility of Chat- GPT for Event Extraction, Mar. 2023. arXiv:2303.03836 [cs] version: 2
arXiv 2023
-
[8]
Y . Guo, Z. Li, X. Jin, Y . Liu, Y . Zeng, W. Liu, X. Li, P. Yang, L. Bai, J. Guo, et al. Retrieval-augmented code generation for universal infor- mation extraction. In CCF International Conference on Natural Lan- guage Processing and Chinese Computing. Springer, 2024
work page 2024
Show all 40 references
-
[9]
Han, I.-H
R. Han, I.-H. Hsu, J. Sun, J. Baylon, Q. Ning, D. Roth, and N. Peng. Ester: A machine reading comprehension dataset for reasoning about event semantic relations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021
2021
-
[10]
Hsu, K.-H
I.-H. Hsu, K.-H. Huang, E. Boschee, S. Miller, P. Natarajan, K.-W. Chang, and N. Peng. DEGREE: A Data-Efficient Generation-Based Event Extraction Model. In NAACL: Human Language Technologies, Seattle, United States, July 2022. Association for Computational Lin- guistics
2022
-
[11]
Huang, R
G. Huang, R. Xu, Y . Zeng, J. Chen, Z. Yang, and W. E. An Iteratively Parallel Generation Method with the Pre-Filling Strategy for Document- level Event Extraction. In H. Bouamor, J. Pino, and K. Bali, editors, EMNLP, Singapore, Dec. 2023. Association for Computational Lin- guistics
2023
-
[12]
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chap- lot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7B, Oct. 2023. arXiv:2310.06825 [cs]
2023 arXiv
-
[13]
Kojima, S
T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa. Large Lan- guage Models are Zero-Shot Reasoners. Advances in Neural Informa- tion Processing Systems, Dec. 2022
2022
-
[14]
F. Li, W. Peng, Y . Chen, Q. Wang, L. Pan, Y . Lyu, and Y . Zhu. Event Extraction as Multi-turn Question Answering. In T. Cohn, Y . He, and Y . Liu, editors,Findings of the Association for Computational Linguis- tics: EMNLP 2020, Online, Nov. 2020. Association for Computationa...
2020
-
[15]
M. Li, A. Zareian, Y . Lin, X. Pan, S. Whitehead, B. Chen, B. Wu, H. Ji, S.-F. Chang, C. V oss, D. Napierski, and M. Freedman. GAIA: A fine- grained multimedia knowledge extraction system. In A. Celikyilmaz and T.-H. Wen, editors, Proceedings of the 58th Annual Meeting of the ...
2020
-
[16]
S. Li, H. Ji, and J. Han. Document-Level Event Argument Extrac- tion by Conditional Generation. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y . Zhou, editors,Proceedings of the 2021 Confer- ence of th...
2021
-
[17]
Y . Lin, H. Ji, F. Huang, and L. Wu. A Joint Neural Model for In- formation Extraction with Global Features. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, ACL, Online, July 2020. Asso- ciation for Computational Linguistics
2020
-
[18]
J. Liu, Y . Chen, K. Liu, W. Bi, and X. Liu. Event Extraction as Machine Reading Comprehension. In B. Webber, T. Cohn, Y . He, and Y . Liu, ed- itors, EMNLP, Online, Nov. 2020. Association for Computational Lin- guistics
2020
-
[19]
X. Liu, H. Huang, G. Shi, and B. Wang. Dynamic Prefix- Tuning for Generative Template-based Event Extraction, May 2022. arXiv:2205.06166 [cs]
2022 arXiv
-
[20]
Y . Lu, H. Lin, J. Xu, X. Han, J. Tang, A. Li, L. Sun, M. Liao, and S. Chen. Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction. In ACL, Online, Aug. 2021. Associa- tion for Computational Linguistics
2021
-
[21]
Y . Lu, Q. Liu, D. Dai, X. Xiao, H. Lin, X. Han, L. Sun, and H. Wu. Unified Structure Generation for Universal Information Extraction. In S. Muresan, P. Nakov, and A. Villavicencio, editors, ACL, Dublin, Ire- land, May 2022. Association for Computational Linguistics
2022
-
[22]
C. Ma, H. Zhao, J. Zhang, J. He, and L. Kong. Non-myopic Generation of Language Models for Reasoning and Planning. 2025
2025
-
[23]
T. H. Nguyen, K. Cho, and R. Grishman. Joint Event Extraction via Re- current Neural Networks. In K. Knight, A. Nenkova, and O. Rambow, editors, NAACL: Human Language Technologies, San Diego, Califor- nia, June 2016. Association for Computational Linguistics
2016
-
[24]
GPT-4 Technical Report, Dec
OpenAI. GPT-4 Technical Report, Dec. 2023. arXiv:2303.08774 [cs]
2023 arXiv
-
[25]
C. Pang, Y . Cao, Q. Ding, and P. Luo. Guideline Learning for In- Context Information Extraction. In H. Bouamor, J. Pino, and K. Bali, editors, EMNLP, Singapore, Dec. 2023. Association for Computational Linguistics
2023
-
[26]
Pouran Ben Veyseh, V
A. Pouran Ben Veyseh, V . Lai, F. Dernoncourt, and T. H. Nguyen. Un- leash GPT-2 Power for Event Detection. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, ACL, Online, Aug. 2021. Association for Computa- tional Linguistics
2021
-
[27]
Touvron, T
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and Efficient Foun- dation Language Models, Feb. 2023. arXiv:2302.13971 [cs]
2023 arXiv
-
[28]
Wadden, U
D. Wadden, U. Wennberg, Y . Luan, and H. Hajishirzi. Entity, Relation, and Event Extraction with Contextualized Span Representations. In K. Inui, J. Jiang, V . Ng, and X. Wan, editors, EMNLP-IJCNLP, Hong Kong, China, Nov. 2019. Association for Computational Linguistics
2019
-
[29]
X. Wang, S. Li, and H. Ji. Code4Struct: Code Generation for Few- Shot Event Structure Prediction. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Toronto, C...
2023
-
[30]
Y . Wang, S. Zhao, Z. Wang, H. Huang, M. Fan, Y . Zhang, Z. Wang, H. Wang, and T. Liu. Strategic Chain-of-Thought: Guid- ing Accurate Reasoning in LLMs through Strategy Elicitation, 2024. arXiv:2409.03271 [cs]
2024 arXiv
-
[31]
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V . Le, and D. Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.Advances in Neural Information Processing Systems, Dec. 2022
2022
-
[32]
X. Wei, X. Cui, N. Cheng, X. Wang, X. Zhang, S. Huang, P. Xie, J. Xu, Y . Chen, M. Zhang, Y . Jiang, and W. Han. Zero-Shot Information Ex- traction via Chatting with ChatGPT, Feb. 2023. arXiv:2302.10205 [cs]
2023 arXiv
-
[33]
Y . Wu, Z. Sun, S. Li, S. Welleck, and Y . Yang. Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem- Solving with Language Models, 2024. arXiv:2408.00724 [cs]
2024 arXiv
-
[34]
N. Xia, H. Yu, Y . Wang, J. Xuan, and X. Luo. DAFS: a domain aware few shot generative model for event detection.Mach. Learn., (3), 2023
2023
-
[35]
D. Xu, W. Chen, W. Peng, C. Zhang, T. Xu, X. Zhao, X. Wu, Y . Zheng, Y . Wang, and E. Chen. Large language models for generative informa- tion extraction: a survey
-
[36]
S. Yang, D. Feng, L. Qiao, Z. Kan, and D. Li. Exploring Pre-trained Language Models for Event Extraction and Generation. In A. Korho- nen, D. Traum, and L. Màrquez, editors,ACL, Florence, Italy, July 2019. Association for Computational Linguistics
2019
-
[37]
Zhang, X
H. Zhang, X. Liu, H. Pan, Y . Song, and C. W.-K. Leung. Aser: A large- scale eventuality knowledge graph. In Proceedings of the web confer- ence 2020, 2020
2020
-
[38]
Zhang, F
Q. Zhang, F. Lyu, Z. Sun, L. Wang, W. Zhang, Z. Guo, Y . Wang, N. Muennighoff, I. King, X. Liu, and C. Ma. What, How, Where, and How Well? A Survey on Test-Time Scaling in Large Language Models,
-
[39]
lent" and
G. Zhao, X. Gong, X. Yang, G. Dong, S. Lu, and S. Li. DemoSG: Demonstration-enhanced Schema-guided Generation for Low-resource Event Extraction. In H. Bouamor, J. Pino, and K. Bali, editors, Find- ings of the Association for Computational Linguistics: EMNLP 2023 , Singapore, D...
2023
-
[2025]
arXiv:2503.24235 [cs]
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.