Pith. sign in

REVIEW 1 cited by

Limitations of Pinned AUC for Measuring Unintended Bias

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1903.02088 v1 pith:2M7DY3O5 submitted 2019-03-05 stat.ML cs.LG

Daniel Borkan , Lucas Dixon , John Li , Jeffrey Sorensen , Nithum Thain , Lucy Vasserman This is my paper · ORCID
classification stat.MLcs.LG
keywords biasmetricpinnedunintendedlimitationsreportbiasescarefully
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

This report examines the Pinned AUC metric introduced and highlights some of its limitations. Pinned AUC provides a threshold-agnostic measure of unintended bias in a classification model, inspired by the ROC-AUC metric. However, as we highlight in this report, there are ways that the metric can obscure different kinds of unintended biases when the underlying class distributions on which bias is being measured are not carefully controlled.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Debiasing Personal Identities in Toxicity Classification

    cs.CL 2019-08 conditional novelty 4.0 of 10

    A toxicity model trained without identity-targeting comments has similar overall AUC but worse per-subgroup accuracy than a model trained on mixed data, especially on false negatives.

Pith tools