RECORD
Extreme Extraction: Only One Hour per Relation
Computation and Language · Submitted 21 Jun 2015
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Experiments show that experts can create quality extractors in under an hour and even NLP novices can author good extractors.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Information Extraction (IE) aims to automatically generate a large knowledge base from natural language text, but progress remains slow. Supervised learning requires copious human annotation, while unsupervised and weakly supervised approaches do not deliver competitive accuracy. As a result, most fielded applications of IE, as well as the leading TAC-KBP systems, rely on significant amounts of manual engineering. Even "Extreme" methods, such as those reported in Freedman et al. 2011, require about 10 hours of expert labor per relation. This paper shows how to reduce that effort by an order of magnitude. We present a novel system, InstaRead, that streamlines authoring with an ensemble of methods: 1) encoding extraction rules in an expressive and compositional representation, 2) guiding the user to promising rules based on corpus statistics and mined resources, and 3) introducing a new interactive development cycle that provides immediate feedback --- even on large datasets. Experiments show that experts can create quality extractors in under an hour and even NLP novices can author good extractors. These extractors equal or outperform ones obtained by comparably supervised and state-of-the-art distantly supervised approaches.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Information Extraction (IE) aims to automatically generate a large knowledge base from natural language text, but progress remains slow. | background 0.99 | 0.06 | 0.02 | asserted 1.00 |
| s1 | Supervised learning requires copious human annotation, while unsupervised and weakly supervised approaches do not deliver competitive accuracy. | background 0.82 | 0.09 | 0.02 | asserted 1.00 |
| s2 | As a result, most fielded applications of IE, as well as the leading TAC-KBP systems, rely on significant amounts of manual engineering. | background 0.96 | 0.08 | 0.02 | asserted 1.00 |
| s3 | Even "Extreme" methods, such as those reported in Freedman et al. 2011, require about 10 hours of expert labor per relation. | background 0.83 | 0.05 | 0.03 | asserted 0.99 |
| s4 | This paper shows how to reduce that effort by an order of magnitude. | contribution 0.86 | 0.74 | 0.52 | asserted 1.00 |
| s5 | We present a novel system, InstaRead, that streamlines authoring with an ensemble of methods: 1) encoding extraction rules in an expressive and compositional representation, 2) guiding the user to promising rules based on corpus statistics and mined resources, and 3) introducing a new interactive development cycle that provides immediate feedback --- even on large datasets. | method 0.74 | 0.90 | 0.09 | asserted 1.00 |
| s6 | Experiments show that experts can create quality extractors in under an hour and even NLP novices can author good extractors. | result 1.00 | 0.95 | 0.55 | asserted 0.99 |
| s7 | These extractors equal or outperform ones obtained by comparably supervised and state-of-the-art distantly supervised approaches. | result 1.00 | 0.90 | 0.24 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Information Extraction (IE) aims to automatically generate a large knowledge base from natural language text, but progress remains slow. Supervised learning requires copious human annotation, while unsupervised and weakly supervised approaches do not deliver competitive accuracy. As a result, most fielded applications of IE, as well as the leading TAC-KBP systems, rely on significant amounts of manual engineering. Even "Extreme" methods, such as those reported in Freedman et al. 2011, require about 10 hours of expert labor per relation. This paper shows how to reduce that effort by an order of magnitude. We present a novel system, InstaRead, that streamlines authoring with an ensemble of methods: 1) encoding extraction rules in an expressive and compositional representation, 2) guiding the user to promising rules based on corpus statistics and mined resources, and 3) introducing a new interactive development cycle that provides immediate feedback --- even on large datasets. Experiments show that experts can create quality extractors in under an hour and even NLP novices can author good extractors. These extractors equal or outperform ones obtained by comparably supervised and state-of-the-art distantly supervised approaches.
Human review
References
How the literature uses it
2 arXiv papers cite this in 3 sentences
Papers citing it · 1 sentence without a confident label
Cites it as support 1 paper
-
The training of individual models has been in effective in other tasks [ref] .
End-to-End Learning for Answering Structured Queries Directly over Text · Related Work · 0.96
Cites it as background 1 paper
-
Text Databases Authors of Deep Dive [ref] , InstaRead [ref] and System-T [ref] propose declarative SQL-based query languages for integrating relational data with text data.
IDEL: In-Database Entity Linking with Neural Embeddings · Related Work · 0.97
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.