CompanionSim provides a synthetic data framework showing that AI companionship behaviors decrease third-party perceptions of likability, humanlikeness, and trust, with significant demographic heterogeneity.
D 3 CODE : Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation
6 Pith papers cite this work, alongside 4 external citations. Polarity classification is still indexing.
6
Pith papers citing it
4
external citations · Crossref
representative citing papers
New Ghost Annotator framework uses conformal prediction to show LLMs of different sizes and families produce labels no human annotator chose and align least with 18-30 male Sub-Saharan African annotators across content moderation datasets.
Large-scale statistical analysis of four harmful language datasets reveals that interactions between annotator characteristics and linguistic cues drive annotation variation, with lexical features and attitudes prominent but patterns varying by dataset.
Directly predicting whether annotators will disagree on a value label outperforms inferring disagreement from per-annotator value predictions on the Touché23-ValueEval dataset.
NUTMEG jointly estimates annotator competence and per-subpopulation ground-truth labels, separating systematic disagreement from spam better than traditional aggregation.
The paper argues that LLMs should be evaluated and built for meta-cultural competence rather than static knowledge of specific cultures, and gives a first, illustrative measurement of one component.
citing papers explorer
-
CompanionSim: Synthetic Data for Evaluating Anthropomorphism in Human-AI Relationships
CompanionSim provides a synthetic data framework showing that AI companionship behaviors decrease third-party perceptions of likability, humanlikeness, and trust, with significant demographic heterogeneity.
-
The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction
New Ghost Annotator framework uses conformal prediction to show LLMs of different sizes and families produce labels no human annotator chose and align least with 18-30 male Sub-Saharan African annotators across content moderation datasets.
-
Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation
Large-scale statistical analysis of four harmful language datasets reveals that interactions between annotator characteristics and linguistic cues drive annotation variation, with lexical features and attitudes prominent but patterns varying by dataset.
-
Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments
Directly predicting whether annotators will disagree on a value label outperforms inferring disagreement from per-annotator value predictions on the Touché23-ValueEval dataset.
-
NUTMEG: Separating Signal From Noise in Annotator Disagreement
NUTMEG jointly estimates annotator competence and per-subpopulation ground-truth labels, separating systematic disagreement from spam better than traditional aggregation.
-
Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness
The paper argues that LLMs should be evaluated and built for meta-cultural competence rather than static knowledge of specific cultures, and gives a first, illustrative measurement of one component.