RECORD
Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research
Computation and Language · Submitted 4 Oct 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our findings also suggest that data duplication has an impact on the current claims of state-of-the-art performance, potentially leading to an overestimation of model effectiveness in real-world scenarios.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Research in natural language processing (NLP) for Computational Social Science (CSS) heavily relies on data from social media platforms. This data plays a crucial role in the development of models for analysing socio-linguistic phenomena within online communities. In this work, we conduct an in-depth examination of 20 datasets extensively used in NLP for CSS to comprehensively examine data quality. Our analysis reveals that social media datasets exhibit varying levels of data duplication. Consequently, this gives rise to challenges like label inconsistencies and data leakage, compromising the reliability of models. Our findings also suggest that data duplication has an impact on the current claims of state-of-the-art performance, potentially leading to an overestimation of model effectiveness in real-world scenarios. Finally, we propose new protocols and best practices for improving dataset development from social media data and its usage.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Research in natural language processing (NLP) for Computational Social Science (CSS) heavily relies on data from social media platforms. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s1 | This data plays a crucial role in the development of models for analysing socio-linguistic phenomena within online communities. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s2 | In this work, we conduct an in-depth examination of 20 datasets extensively used in NLP for CSS to comprehensively examine data quality. | contribution 0.79 | 0.96 | 0.04 | asserted 0.97 |
| s3 | Our analysis reveals that social media datasets exhibit varying levels of data duplication. | result 1.00 | 0.94 | 0.18 | asserted 1.00 |
| s4 | Consequently, this gives rise to challenges like label inconsistencies and data leakage, compromising the reliability of models. | interpretation 0.84 | 0.37 | 0.17 | asserted 0.99 |
| s5 | Our findings also suggest that data duplication has an impact on the current claims of state-of-the-art performance, potentially leading to an overestimation of model effectiveness in real-world scenarios. | interpretation 0.90 | 0.89 | 0.42 | hedged possibility 1.00 |
| s6 | Finally, we propose new protocols and best practices for improving dataset development from social media data and its usage. | contribution 0.81 | 0.71 | 0.18 | asserted 0.98 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Research in natural language processing (NLP) for Computational Social Science (CSS) heavily relies on data from social media platforms. This data plays a crucial role in the development of models for analysing socio-linguistic phenomena within online communities. In this work, we conduct an in-depth examination of 20 datasets extensively used in NLP for CSS to comprehensively examine data quality. Our analysis reveals that social media datasets exhibit varying levels of data duplication. Consequently, this gives rise to challenges like label inconsistencies and data leakage, compromising the reliability of models. Our findings also suggest that data duplication has an impact on the current claims of state-of-the-art performance, potentially leading to an overestimation of model effectiveness in real-world scenarios. Finally, we propose new protocols and best practices for improving dataset development from social media data and its usage.
Human review
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.