Familiarity quantifies semantic overlap between training and evaluation entity types in zero-shot NER, showing that much of the performance gains in current synthetic-data models can be explained by label leakage.
The CaLiGraph Ontology as a Challenge for OWL Reasoners
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
CaLiGraph is a large-scale cross-domain knowledge graph generated from Wikipedia by exploiting the category system, list pages, and other list structures in Wikipedia, containing more than 15 million typed entities and around 10 million relation assertions. Other than knowledge graphs such as DBpedia and YAGO, whose ontologies are comparably simplistic, CaLiGraph also has a rich ontology, comprising more than 200,000 class restrictions. Those two properties - a large A-box and a rich ontology - make it an interesting challenge for benchmarking reasoners. In this paper, we show that a reasoning task which is particularly relevant for CaLiGraph, i.e., the materialization of owl:hasValue constraints into assertions between individuals and between individuals and literals, is insufficiently supported by available reasoning systems. We provide differently sized benchmark subsets of CaLiGraph, which can be used for performance analysis of reasoning systems.
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Familiarity: Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data
Familiarity quantifies semantic overlap between training and evaluation entity types in zero-shot NER, showing that much of the performance gains in current synthetic-data models can be explained by label leakage.