Zero-shot classifiers built on sentence-transformer language models (T5-xl, MiniLM) outperform much larger LLMs on multi-label, hierarchical requirements classification, and a hop-count label distance metric helps rank models.
EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Events serve as fundamental units of occurrence within various contexts. The processing of event semantics in textual information forms the basis of numerous natural language processing (NLP) applications. Recent studies have begun leveraging large language models (LLMs) to address event semantic processing. However, the extent that LLMs can effectively tackle these challenges remains uncertain. Furthermore, the lack of a comprehensive evaluation framework for event semantic processing poses a significant challenge in evaluating these capabilities. In this paper, we propose an overarching framework for event semantic processing, encompassing understanding, reasoning, and prediction, along with their fine-grained aspects. To comprehensively evaluate the event semantic processing abilities of models, we introduce a novel benchmark called EVEVAL. We collect 8 datasets that cover all aspects of event semantic processing. Extensive experiments are conducted on EVEVAL, leading to several noteworthy findings based on the obtained results.
fields
cs.SE 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Language Models to Support Multi-Label Classification of Industrial Data
Zero-shot classifiers built on sentence-transformer language models (T5-xl, MiniLM) outperform much larger LLMs on multi-label, hierarchical requirements classification, and a hop-count label distance metric helps rank models.