Pith. sign in

REVIEW 5 cited by

Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2112.04554 v1 pith:QNFHGMTM submitted 2021-12-08 cs.LG

Remi Denton , Mark Díaz , Ian Kivlichan , Vinodkumar Prabhakaran , Rachel Rosen This is my paper · ORCID
classification cs.LG
keywords datasetannotatorsconsiderationsannotationannotationsaroundethicalinsights
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Human annotations play a crucial role in machine learning (ML) research and development. However, the ethical considerations around the processes and decisions that go into building ML datasets has not received nearly enough attention. In this paper, we survey an array of literature that provides insights into ethical considerations around crowdsourced dataset annotation. We synthesize these insights, and lay out the challenges in this space along two layers: (1) who the annotator is, and how the annotators' lived experiences can impact their annotations, and (2) the relationship between the annotators and the crowdsourcing platforms and what that relationship affords them. Finally, we put forth a concrete set of recommendations and considerations for dataset developers at various stages of the ML data pipeline: task formulation, selection of annotators, platform and infrastructure choices, dataset analysis and evaluation, and dataset documentation and release.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Small hyperbolic models (146M–3B) report 100% creative-seed preference, 90.7% compliance-gap detection, and a selective-gating skeleton–wallpaper memory pilot as a companion-AI stack.

  2. On the Sensitivity of Instruction-tuned LLMs to Harmful Sentences in Long Inputs

    cs.CL 2025-10 conditional novelty 6.0 of 10

    LLMs detect harmful sentences in long inputs best at around 25% prevalence and when placed early, and worse when sparse, late, or implicit.

  3. Characterizing Network Structure of Anti-Trans Actors on TikTok

    cs.HC 2025-01 reject novelty 5.0 of 10

    Using a new taxonomy and a RAG-enhanced LLM classifier on TikTok, this paper finds anti-trans accounts outnumber and heavily interact with pro-trans accounts, but the classifier's evaluation is not shown to be leak-free.

  4. The Human Labour of Data Work: Capturing Cultural Diversity through World Wide Dishes

    cs.CY 2025-02 conditional novelty 4.0 of 10

    A design retrospective of World Wide Dishes identifies three dimensions of community ambassador labor, trust building, accessibility, and cultural contextualization, as essential to participatory dataset creation.

  5. A Critical Field Guide for Working with Machine Learning Datasets

    cs.CY 2025-01 unverdicted novelty 2.0 of 10

    A field guide from the Knowing Machines project that turns existing critical dataset studies scholarship into lifecycle questions for practitioners, with no new empirical or formal results.

Pith tools