NLP papers commonly report annotator recruitment, expertise, and volume but frequently omit training, compensation, socio-demographics, adjudication, and agreement metrics, with reporting improving over time yet remaining uneven across tasks and venues.
Preprint, arXiv:2505.24712
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2026 1verdicts
UNVERDICTED 1representative citing papers
citing papers explorer
-
Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025
NLP papers commonly report annotator recruitment, expertise, and volume but frequently omit training, compensation, socio-demographics, adjudication, and agreement metrics, with reporting improving over time yet remaining uneven across tasks and venues.