Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction

Charles Jochim; Debasis Ganguly; Francesca Bonin; Martin Gleize; Yufang Hou

arxiv: 1906.09317 · v1 · pith:MVBCTSRSnew · submitted 2019-06-21 · 💻 cs.CL · cs.AI

Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction

Yufang Hou , Charles Jochim , Martin Gleize , Francesca Bonin , Debasis Ganguly This is my paper

Pith reviewed 2026-05-25 18:35 UTC · model grok-4.3

classification 💻 cs.CL cs.AI

keywords information extractionNLP leaderboardsscientific summarizationtask dataset metric scorenatural language processingautomatic evaluationresearch tracking

0 comments

The pith

A framework extracts tasks, datasets, metrics and scores from NLP papers to enable automatic leaderboard construction.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tackles the growing difficulty of tracking results across the expanding set of NLP tasks and datasets by building an automated extraction system. The authors created two dedicated datasets and introduced the TDMS-IE framework to pull out task-dataset-metric-score tuples directly from research papers. Experiments show the model beats several baselines by a large margin. If the approach holds, it would let the community maintain up-to-date leaderboards without relying solely on manual curation as publication volume increases.

Core claim

The authors build two datasets and develop a framework (TDMS-IE) aimed at automatically extracting task, dataset, metric and score from NLP papers, towards the automatic construction of leaderboards. Experiments show that their model outperforms several baselines by a large margin. Their model is a first step towards automatic leaderboard construction, e.g., in the NLP domain.

What carries the argument

TDMS-IE framework for extracting task-dataset-metric-score tuples from scientific papers.

Load-bearing premise

The two newly created datasets are representative enough of real NLP papers and the extracted tuples can be directly used to construct accurate leaderboards without substantial human verification.

What would settle it

Manual expert review of leaderboards built from the model's extractions on a fresh collection of NLP papers reveals frequent incorrect or incomplete task-dataset-metric-score tuples.

Figures

Figures reproduced from arXiv: 1906.09317 by Charles Jochim, Debasis Ganguly, Francesca Bonin, Martin Gleize, Yufang Hou.

**Figure 1.** Figure 1: An illustrative example of leaderboard construction from a sample article. The cue words related to the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Figure 2: Examples of document representation (DocTAET) and score context (SC) representation. Entails H: Named entity recognition, ACE 2005 (Test), Accuracy H: Entity linking, ACE 2005 (Test), Accuracy H: Coreference resolution, CoNLL 2012 (Test), Avg. F1 Document representation H: Dependency parsing, Penn Treebank, UAS H: … Hypothesis space: <Task, Dataset, Metric> triples from a taxonomy Score1: s1 context Hypoth… view at source ↗

**Figure 3.** Figure 3: System architecture for TDMS-IE. score context suggests a DM pair. Finally, for each predicted TDM triple, we select the score whose context has the highest confidence in predicting a link to the constituent DM pair. 5 Experimental Setup 5.1 Training/Test Datasets We split NLP-TDMS (described in Section 3) into training and test sets. The partitioning ensures that every TDM triple annotated in NLP-TDMS ap… view at source ↗

read the original abstract

While the fast-paced inception of novel tasks and new datasets helps foster active research in a community towards interesting directions, keeping track of the abundance of research activity in different areas on different datasets is likely to become increasingly difficult. The community could greatly benefit from an automatic system able to summarize scientific results, e.g., in the form of a leaderboard. In this paper we build two datasets and develop a framework (TDMS-IE) aimed at automatically extracting task, dataset, metric and score from NLP papers, towards the automatic construction of leaderboards. Experiments show that our model outperforms several baselines by a large margin. Our model is a first step towards automatic leaderboard construction, e.g., in the NLP domain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper releases two new datasets and a TDMS-IE model for pulling task-dataset-metric-score tuples from NLP papers, but the experiments stay inside the authors' own annotations and never check whether the output would produce usable leaderboards.

read the letter

The one thing to take away is that the authors built two fresh datasets and a dedicated extraction system aimed at automatic NLP leaderboards, then showed their model beats baselines on those sets. That combination is the actual new piece rather than a routine IE extension. The framing around tracking empirical progress is also direct and practical, which is better than papers that just add another layer to existing scientific IE without a clear use case. They get credit for releasing the data and for targeting a real pain point in the field. The evaluation, however, stays narrow. All reported gains come from the datasets the authors annotated themselves, with no inter-annotator agreement numbers, no comparison to existing leaderboards, and no end-to-end test of whether the extracted tuples would require heavy human fixes before they could be trusted. The stress-test note is right on this: the margin could simply reflect the authors' labeling conventions rather than extraction quality that generalizes. Without those checks, the claim that this advances automatic leaderboard construction rests on an untested assumption about representativeness and downstream fidelity. The work is aimed at people who build tools for scientific information extraction or who maintain leaderboards in fast-moving areas like NLP. A reader who needs new annotated data for tuple extraction tasks would get concrete value from the resources. It is worth sending to peer review because the datasets are new and the goal is stated clearly; referees can then press on annotation quality and ask for the missing validation steps. I would not cite it in its current form, but the resources alone justify a proper review.

Referee Report

3 major / 1 minor

Summary. The manuscript introduces two new datasets for extracting (task, dataset, metric, score) tuples from NLP papers and presents the TDMS-IE framework for this extraction task. Experiments on the author-created datasets show that TDMS-IE outperforms several baselines by a large margin, positioned as an initial step toward automatic scientific leaderboard construction.

Significance. If the extraction quality generalizes and the tuples prove sufficiently accurate for leaderboard use, the work could reduce manual effort in tracking NLP results. The current evaluation, however, provides no evidence that the reported gains support deployable leaderboards without substantial human correction.

major comments (3)

[Dataset construction] Dataset construction: no inter-annotator agreement is reported for either of the two new datasets, leaving the reliability of the gold annotations used to train and evaluate TDMS-IE unquantified.
[Experiments] Experiments: the evaluation contains no end-to-end test measuring how accurately the extracted tuples reconstruct leaderboards on held-out papers; the large-margin claim therefore does not yet demonstrate that downstream leaderboards would require only minimal manual verification.
[Abstract and Experiments] Abstract and Experiments: performance is reported exclusively on author-annotated data with no external validation set or papers using different result-reporting conventions, so it remains unclear whether the margin reflects genuine extraction robustness rather than annotation conventions specific to the authors.

minor comments (1)

[Abstract] The abstract would benefit from stating the sizes and annotation guidelines of the two datasets.

Simulated Author's Rebuttal

3 responses · 0 unresolved

Thank you for the constructive feedback on our manuscript. We address each major comment below, indicating planned revisions where appropriate.

read point-by-point responses

Referee: Dataset construction: no inter-annotator agreement is reported for either of the two new datasets, leaving the reliability of the gold annotations used to train and evaluate TDMS-IE unquantified.

Authors: We agree that inter-annotator agreement should be reported to quantify annotation reliability. We will add a second annotator for a subset of the papers, compute agreement metrics such as Cohen's kappa, and include these results in the revised manuscript. revision: yes
Referee: Experiments: the evaluation contains no end-to-end test measuring how accurately the extracted tuples reconstruct leaderboards on held-out papers; the large-margin claim therefore does not yet demonstrate that downstream leaderboards would require only minimal manual verification.

Authors: The paper explicitly frames TDMS-IE as an initial step and does not claim the results support fully deployable leaderboards without human oversight. The evaluation targets extraction accuracy. We will add a limitations discussion on the absence of end-to-end leaderboard reconstruction and note that such an evaluation would require further annotation effort beyond the current scope. revision: partial
Referee: Abstract and Experiments: performance is reported exclusively on author-annotated data with no external validation set or papers using different result-reporting conventions, so it remains unclear whether the margin reflects genuine extraction robustness rather than annotation conventions specific to the authors.

Authors: The datasets follow a documented annotation protocol, and evaluation uses held-out papers from the same collection. External validation on independently created data is not available for these new resources. We will revise the abstract and add a limitations section clarifying the evaluation scope and the potential influence of annotation conventions. revision: partial

Circularity Check

0 steps flagged

No circularity in derivation chain or experimental claims

full rationale

The paper describes construction of two author-annotated datasets and development of an information extraction model (TDMS-IE) that is evaluated against baselines on those datasets. No equations, fitted parameters presented as predictions, self-citation load-bearing arguments, uniqueness theorems, or ansatzes are present. The central performance claim rests on standard supervised evaluation rather than any reduction to self-defined inputs by construction. The work is a self-contained applied extraction task with independent model and annotation content.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only; no mathematical derivations, free parameters, or invented entities are mentioned. The work relies on standard supervised learning assumptions for information extraction.

pith-pipeline@v0.9.0 · 5657 in / 964 out tokens · 16055 ms · 2026-05-25T18:35:14.755583+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

work page
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

work page
[3]

Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Wang, Chris Willhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni. 2018. ht...

work page 2018
[4]

Waleed Ammar, Matthew Peters, Chandra Bhagavatula, and Russell Power. 2017. https://doi.org/10.18653/v1/S17-2097 The AI2 system at SemEval-2017 Task 10 ( ScienceIE ): Semi-supervised end-to-end entity and relation extraction . In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, 3 -- 4 August 2017 , p...

work page doi:10.18653/v1/s17-2097 2017
[5]

Armstrong, Alistair Moffat, William Webber, and Justin Zobel

Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Improvements that don't add up: Ad-hoc retrieval results since 1998. In Proceedings of the ACM 18th Conference on Information and Knowledge Management (CIKM 2009), Hong Kong, China, 2--6 November 2009 , pages 601--610

work page 2009
[6]

Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N., Kaiser Lukasz, and Polosukhin Illia. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NIPS 2017), pages 1--11

work page 2017
[7]

Awais Athar and Simone Teufel. 2012 a . http://www.aclweb.org/anthology/N12-1073 Context-enhanced citation sentiment detection . In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Montr \' e al, Qu \' e bec, Canada, 3--8 June 2012 , pages 597--601

work page 2012
[8]

Awais Athar and Simone Teufel. 2012 b . http://www.aclweb.org/anthology/W12-4303 Detection of implicit citations for sentiment detection . In Proceedings of the Workshop on Detecting Structure in Scholarly Discourse, Jeju Island, Republic of Korea, 12 July , pages 18--26

work page 2012
[9]

Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. http://aclweb.org/anthology/S17-2091 SemEval 2017 Task 10: ScienceIE - E xtracting keyphrases and relations from scientific publications . In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, 3 -- 4 August ...

work page 2017
[10]

Isabelle Augenstein and Anders S gaard. 2017. https://doi.org/10.18653/v1/P17-2054 Multi-task learning of keyphrase boundary classification . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vancouver, Canada, 30 July -- 4 August 2017 , pages 341--346

work page doi:10.18653/v1/p17-2054 2017
[11]

Steven Bird, Robert Dale, Bonnie Dorr, Bryan Gibson, Mark Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir Radev, and Yee Fan Tan. 2008. The ACL anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. In Proceedings of the 6th International Conference on Language Resources and Evaluation, Marrakech,...

work page 2008
[12]

Bowman, Gabor Angeli, Christopher Potts, and Christopher D

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, 17--21 September 2015 , pages 632--642

work page 2015
[13]

Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, pages 177--190, Heidelberg, Germany

work page 2006
[14]

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. 2014. http://papers.nips.cc/paper/5258-saga-a-fast-incremental-gradient-method-with-support-for-non-strongly-convex-composite-objectives.pdf SAGA : A fast incremental gradient method with support for non-strongly convex composite objectives . In Advances in Neural Information Processing Systems 27 (NI...

work page 2014
[15]

Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, USA, 2--7 June 2019 , pages 4171--4186

work page 2019
[16]

Kathrin Eichler, Feiyu Xu, Hans Uszkoreit, and Sebastian Krause. 2017. https://doi.org/10.18653/v1/S17-1026 Generating pattern-based entailment graphs for relation extraction . In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017), Vancouver, Canada, 3 -- 4 August 2017 , pages 220--229

work page doi:10.18653/v1/s17-1026 2017
[17]

Nicola Ferro, Norbert Fuhr, and Andreas Rauber. 2018. https://doi.org/10.1145/3268408 Introduction to the special issue on reproducibility in information retrieval: Evaluation campaigns, collections, and analyses . Journal of Data and Information Quality, 10(3):9:1--9:4

work page doi:10.1145/3268408 2018
[18]

Kata G \' a bor, Davide Buscaldi, Anne - Kathrin Schumann, Behrang QasemiZadeh, Ha \" fa Zargayouna, and Thierry Charnois. 2018. https://aclanthology.info/papers/S18-1111/s18-1111 Semeval-2018 task 7: Semantic relation extraction and classification in scientific papers . In Proceedings of The 12th International Workshop on Semantic Evaluation, SemEval@NAA...

work page 2018
[19]

Sonal Gupta and Christopher Manning. 2011. Analyzing the dynamics of research by extracting key aspects of scientific papers. In Proceedings of 5th international joint conference on natural language processing, Chiang Mai, Thailand, 8--13 November 2011 , pages 1--9

work page 2011
[20]

Kata Gábor, Haifa Zargayouna, Davide Buscaldi, Isabelle Tellier, and Thierry Charnois. 2016. Semantic annotation of the ACL anthology corpus for the automatic analysis of scientific literature. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia, 23--28 May 2016

work page 2016
[21]

David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018. https://doi.org/10.1162/tacl_a_00028 Measuring the evolution of a scientific field through citation frames . Transactions of the Association for Computational Linguistics, 6:391--406

work page doi:10.1162/tacl_a_00028 2018
[22]

Seonhoon Kim, Jin - Hyuk Hong, Inho Kang, and Nojun Kwak. 2019. Semantic sentence matching with densely-connected recurrent and co-attentive information. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Hawaii, USA, 27 January--1 February 2019

work page 2019
[23]

Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 28 July--2 August 2019

work page 2019
[24]

Patrice Lopez. 2009. GROBID: combining automatic bibliographic data recognition and term extraction for scholarship publications. In The 13th European Conference on Digital Libraries ( ECDL 2009), Corfu, Greece, 27 September 27 -- 2 October, 2009 , pages 473--474

work page 2009
[25]

Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. http://aclweb.org/anthology/D18-1360 Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October-- 4 November 2018 , ...

work page 2018
[26]

Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2017. https://doi.org/10.18653/v1/D17-1279 Scientific information extraction with semi-supervised neural tagging . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, 7--11 November 2017 , pages 2641--2651

work page doi:10.18653/v1/d17-1279 2017
[27]

Abiola Obamuyide and Andreas Vlachos. 2018. https://www.aclweb.org/anthology/W18-5511 Zero-shot relation classification as textual entailment . In Proceedings of the First Workshop on Fact Extraction and VER ification ( FEVER ), Brussels, Belgium, 1 November 2018 , pages 72--78

work page 2018
[28]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine Learning Research, 12:2825--2830

work page 2011
[29]

Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme

Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October-- 4 November 2...

work page 2018
[30]

Chen-Tse Tsai, Gourab Kundu, and Dan Roth. 2013. Concept-based analysis of scientific literature. In Proceedings of the ACM 22nd Conference on Information and Knowledge Management (CIKM 2013), San Francisco, California, 27 October--1 November 2013 , pages 1733--1738

work page 2013
[31]

Adam Vogel and Dan Jurafsky. 2012. http://aclweb.org/anthology/W12-3204 He said, she said: Gender in the acl anthology . In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, Jeju Island, Republic of Korea, 10 July , pages 33--41

work page 2012
[32]

Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. https://www.aclweb.org/anthology/I17-1100 Inference is everything: Recasting semantic resources into a unified evaluation framework . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (IJCNLP 2017), Taipei, Taiwan, 27 November -- 1 Dec...

work page 2017

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

work page

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

work page

[3] [3]

Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Wang, Chris Willhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni. 2018. ht...

work page 2018

[4] [4]

Waleed Ammar, Matthew Peters, Chandra Bhagavatula, and Russell Power. 2017. https://doi.org/10.18653/v1/S17-2097 The AI2 system at SemEval-2017 Task 10 ( ScienceIE ): Semi-supervised end-to-end entity and relation extraction . In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, 3 -- 4 August 2017 , p...

work page doi:10.18653/v1/s17-2097 2017

[5] [5]

Armstrong, Alistair Moffat, William Webber, and Justin Zobel

Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Improvements that don't add up: Ad-hoc retrieval results since 1998. In Proceedings of the ACM 18th Conference on Information and Knowledge Management (CIKM 2009), Hong Kong, China, 2--6 November 2009 , pages 601--610

work page 2009

[6] [6]

Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N., Kaiser Lukasz, and Polosukhin Illia. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NIPS 2017), pages 1--11

work page 2017

[7] [7]

Awais Athar and Simone Teufel. 2012 a . http://www.aclweb.org/anthology/N12-1073 Context-enhanced citation sentiment detection . In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Montr \' e al, Qu \' e bec, Canada, 3--8 June 2012 , pages 597--601

work page 2012

[8] [8]

Awais Athar and Simone Teufel. 2012 b . http://www.aclweb.org/anthology/W12-4303 Detection of implicit citations for sentiment detection . In Proceedings of the Workshop on Detecting Structure in Scholarly Discourse, Jeju Island, Republic of Korea, 12 July , pages 18--26

work page 2012

[9] [9]

Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. http://aclweb.org/anthology/S17-2091 SemEval 2017 Task 10: ScienceIE - E xtracting keyphrases and relations from scientific publications . In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, 3 -- 4 August ...

work page 2017

[10] [10]

Isabelle Augenstein and Anders S gaard. 2017. https://doi.org/10.18653/v1/P17-2054 Multi-task learning of keyphrase boundary classification . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vancouver, Canada, 30 July -- 4 August 2017 , pages 341--346

work page doi:10.18653/v1/p17-2054 2017

[11] [11]

Steven Bird, Robert Dale, Bonnie Dorr, Bryan Gibson, Mark Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir Radev, and Yee Fan Tan. 2008. The ACL anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. In Proceedings of the 6th International Conference on Language Resources and Evaluation, Marrakech,...

work page 2008

[12] [12]

Bowman, Gabor Angeli, Christopher Potts, and Christopher D

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, 17--21 September 2015 , pages 632--642

work page 2015

[13] [13]

Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, pages 177--190, Heidelberg, Germany

work page 2006

[14] [14]

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. 2014. http://papers.nips.cc/paper/5258-saga-a-fast-incremental-gradient-method-with-support-for-non-strongly-convex-composite-objectives.pdf SAGA : A fast incremental gradient method with support for non-strongly convex composite objectives . In Advances in Neural Information Processing Systems 27 (NI...

work page 2014

[15] [15]

Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, USA, 2--7 June 2019 , pages 4171--4186

work page 2019

[16] [16]

Kathrin Eichler, Feiyu Xu, Hans Uszkoreit, and Sebastian Krause. 2017. https://doi.org/10.18653/v1/S17-1026 Generating pattern-based entailment graphs for relation extraction . In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017), Vancouver, Canada, 3 -- 4 August 2017 , pages 220--229

work page doi:10.18653/v1/s17-1026 2017

[17] [17]

Nicola Ferro, Norbert Fuhr, and Andreas Rauber. 2018. https://doi.org/10.1145/3268408 Introduction to the special issue on reproducibility in information retrieval: Evaluation campaigns, collections, and analyses . Journal of Data and Information Quality, 10(3):9:1--9:4

work page doi:10.1145/3268408 2018

[18] [18]

Kata G \' a bor, Davide Buscaldi, Anne - Kathrin Schumann, Behrang QasemiZadeh, Ha \" fa Zargayouna, and Thierry Charnois. 2018. https://aclanthology.info/papers/S18-1111/s18-1111 Semeval-2018 task 7: Semantic relation extraction and classification in scientific papers . In Proceedings of The 12th International Workshop on Semantic Evaluation, SemEval@NAA...

work page 2018

[19] [19]

Sonal Gupta and Christopher Manning. 2011. Analyzing the dynamics of research by extracting key aspects of scientific papers. In Proceedings of 5th international joint conference on natural language processing, Chiang Mai, Thailand, 8--13 November 2011 , pages 1--9

work page 2011

[20] [20]

Kata Gábor, Haifa Zargayouna, Davide Buscaldi, Isabelle Tellier, and Thierry Charnois. 2016. Semantic annotation of the ACL anthology corpus for the automatic analysis of scientific literature. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia, 23--28 May 2016

work page 2016

[21] [21]

David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018. https://doi.org/10.1162/tacl_a_00028 Measuring the evolution of a scientific field through citation frames . Transactions of the Association for Computational Linguistics, 6:391--406

work page doi:10.1162/tacl_a_00028 2018

[22] [22]

Seonhoon Kim, Jin - Hyuk Hong, Inho Kang, and Nojun Kwak. 2019. Semantic sentence matching with densely-connected recurrent and co-attentive information. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Hawaii, USA, 27 January--1 February 2019

work page 2019

[23] [23]

Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 28 July--2 August 2019

work page 2019

[24] [24]

Patrice Lopez. 2009. GROBID: combining automatic bibliographic data recognition and term extraction for scholarship publications. In The 13th European Conference on Digital Libraries ( ECDL 2009), Corfu, Greece, 27 September 27 -- 2 October, 2009 , pages 473--474

work page 2009

[25] [25]

Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. http://aclweb.org/anthology/D18-1360 Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October-- 4 November 2018 , ...

work page 2018

[26] [26]

Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2017. https://doi.org/10.18653/v1/D17-1279 Scientific information extraction with semi-supervised neural tagging . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, 7--11 November 2017 , pages 2641--2651

work page doi:10.18653/v1/d17-1279 2017

[27] [27]

Abiola Obamuyide and Andreas Vlachos. 2018. https://www.aclweb.org/anthology/W18-5511 Zero-shot relation classification as textual entailment . In Proceedings of the First Workshop on Fact Extraction and VER ification ( FEVER ), Brussels, Belgium, 1 November 2018 , pages 72--78

work page 2018

[28] [28]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine Learning Research, 12:2825--2830

work page 2011

[29] [29]

Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme

Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October-- 4 November 2...

work page 2018

[30] [30]

Chen-Tse Tsai, Gourab Kundu, and Dan Roth. 2013. Concept-based analysis of scientific literature. In Proceedings of the ACM 22nd Conference on Information and Knowledge Management (CIKM 2013), San Francisco, California, 27 October--1 November 2013 , pages 1733--1738

work page 2013

[31] [31]

Adam Vogel and Dan Jurafsky. 2012. http://aclweb.org/anthology/W12-3204 He said, she said: Gender in the acl anthology . In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, Jeju Island, Republic of Korea, 10 July , pages 33--41

work page 2012

[32] [32]

Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. https://www.aclweb.org/anthology/I17-1100 Inference is everything: Recasting semantic resources into a unified evaluation framework . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (IJCNLP 2017), Taipei, Taiwan, 27 November -- 1 Dec...

work page 2017