REVIEW 8 cited by
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks
read the original abstract
Supervised classification heavily depends on datasets annotated by humans. However, in subjective tasks such as toxicity classification, these annotations often exhibit low agreement among raters. Annotations have commonly been aggregated by employing methods like majority voting to determine a single ground truth label. In subjective tasks, aggregating labels will result in biased labeling and, consequently, biased models that can overlook minority opinions. Previous studies have shed light on the pitfalls of label aggregation and have introduced a handful of practical approaches to tackle this issue. Recently proposed multi-annotator models, which predict labels individually per annotator, are vulnerable to under-determination for annotators with few samples. This problem is exacerbated in crowdsourced datasets. In this work, we propose \textbf{Annotator Aware Representations for Texts (AART)} for subjective classification tasks. Our approach involves learning representations of annotators, allowing for exploration of annotation behaviors. We show the improvement of our method on metrics that assess the performance on capturing individual annotators' perspectives. Additionally, we demonstrate fairness metrics to evaluate our model's equability of performance for marginalized annotators compared to others.
Forward citations
Cited by 8 Pith papers
-
Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins
Small hyperbolic models (146M–3B) report 100% creative-seed preference, 90.7% compliance-gap detection, and a selective-gating skeleton–wallpaper memory pilot as a companion-AI stack.
-
Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins
On the same 720 replies, scoring exposure versus manifestation shifts the auditor-judge gap by ~0.2 AUROC and can reverse their ranking, so single detection AUROCs are under-specified.
-
Negative Ontology of True Target for Machine Learning: Towards Evaluation and Learning under Democratic Supervision
By adopting a negative ontology where the true target does not objectively exist, the paper defines Democratic Supervision and derives the EL-MIATTs framework for ML evaluation and learning with Multiple Inaccurate Tr...
-
Empathy Applicability Modeling for General Health Queries
General health queries can be labeled in advance for whether they call for emotional reactions or interpretive empathy, and classifiers trained on these labels beat simple baselines.
-
Negative Ontology of True Target for Machine Learning: Towards Evaluation and Learning under Democratic Supervision
The paper posits that the true target does not exist and introduces the EL-MIATTs framework for evaluation and learning under Democratic Supervision in machine learning.
-
Negative Ontology of True Target for Machine Learning: Towards Evaluation and Learning under Democratic Supervision
The true target does not objectively exist in ML, so models should use multiple inaccurate true targets under democratic supervision via the EL-MIATTs framework for evaluation and learning.
-
Negative Ontology of True Target for Machine Learning: Towards Evaluation and Learning under Democratic Supervision
Proposes the EL-MIATTs framework for ML predictive modeling by assuming the true target does not exist and defining democratic supervision via multiple inaccurate true targets.
-
Negative Ontology of True Target for Machine Learning: Towards Evaluation and Learning under Democratic Supervision
By assuming the true target does not exist, this paper proposes Democratic Supervision with Multiple Inaccurate True Targets (MIATTs), packaging prior author work into the EL-MIATTs framework.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.