Pith. sign in

REVIEW 5 major objections 8 minor 159 references

A class-conditional Rasch model recovers rare labels in crowdsourcing better than standard aggregators by letting annotator skill and item difficulty differ by class.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 10:17 UTC pith:DET53S7X

load-bearing objection Solid methods paper that fills the DS–GLAD gap for minority recall; the F1 drop on high-recall wins is real and under-discussed, but the operational claim is still coherent. the 5 major comments →

arxiv 2607.24622 v1 pith:DET53S7X submitted 2026-07-27 stat.ML cs.LG

A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detection

Gabriel Singer , Samuel Gruffaz , Olivier Vo Van , Nicolas Vayatis , Argyris Kalogeratos This is my paper
classification stat.ML cs.LG
keywords crowdsourcinglabel aggregationclass imbalanceminority recallRasch modelannotator competenceitem difficultyCondorcet jury theorem
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When the labels that matter most are also the rarest, majority vote and common aggregation methods can look accurate overall while missing the minority class. This paper argues that the missing piece is a model in which both annotator competence and item difficulty are allowed to depend on the true class, so the crowd can contain majority specialists, minority specialists, and people who are good or bad on both. It introduces CC-Rasch, a generative model that does exactly that, proves a Condorcet-style guarantee and a class-proportion preservation result for majority vote under imbalance, and shows on 33 real crowdsourcing datasets that the model consistently leads in minority recall while staying competitive on balanced accuracy. The practical claim is simple: if rare-label recovery is the goal, class-dependent ability and difficulty should be estimated jointly rather than ignored or collapsed into a single skill score.

Core claim

CC-Rasch, which models correctness probability as a logistic of class-specific annotator ability minus class-specific item difficulty (with mixed-effect deviations and uniform error spreading over wrong classes), recovers minority labels more reliably than majority vote, Dawid–Skene, GLAD, CrowdFM, and PLAT across a large real-world benchmark, while remaining competitive in balanced accuracy.

What carries the argument

CC-Rasch: a class-conditional Rasch-style generative model where Z_ir | Y_i=k ~ Bernoulli(σ(α_{r,k}−β_{i,k})), with α_{r,k}=μ_{α,k}+g_{r,k} and β_{i,k}=μ_{β,k}+h_{i,k}, sum-to-zero constraints, Gaussian priors, and EM estimation; it is the object that jointly captures class-dependent competence and difficulty.

Load-bearing premise

Given the true class, every wrong label is treated as equally likely, and correctness is fully summarized by one logistic skill-minus-difficulty number per class.

What would settle it

On a held-out imbalanced crowdsourcing task where confusions are highly asymmetric (for example systematic majority-to-minority bias) or difficulty is not scalar per class, check whether CC-Rasch still leads minority recall against Dawid–Skene and PLAT; a clear loss under those conditions would undercut the model’s central claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Inspection and quality systems that care most about rare defects can prioritize CC-Rasch-style aggregation when only crowd labels are available.
  • Majority vote remains consistent for every class, including minorities, only if each class has enough specialists relative to weak or adversarial annotators.
  • As the number of labels grows, majority vote’s implied class balance converges to the true balance under the paper’s crowd conditions.
  • Gold labels are not always required for strong minority recall: the unsupervised class-conditional model can beat gold-supervised baselines as imbalance grows.
  • Benchmarking label aggregation under imbalance should treat minority recall as primary rather than overall accuracy alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If minority specialists are scarce in a deployed crowd, the paper’s synthetic sweep suggests actively recruiting or weighting them may matter more than adding generic annotators.
  • The uniform-error assumption points to a natural next model: class-to-class confusion structure combined with class-specific difficulty.
  • The same mixed-effects split could transfer to other noisy-label settings where both rater skill and case hardness are known to be label-dependent, such as medical triage or content moderation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper studies label aggregation under class imbalance, where rare classes are the operationally important ones. It makes three contributions: (1) a class-imbalanced Condorcet Jury Theorem (Theorem 3.1) giving crowd-composition conditions under which majority vote asymptotically recovers every class, plus a corollary that MV asymptotically preserves the class proportions; (2) CC-Rasch, a generative model with class-dependent annotator abilities α_{r,k} = μ_{α,k} + g_{r,k} and class-dependent item difficulties β_{i,k} = μ_{β,k} + h_{i,k}, Gaussian priors on the deviations, sum-to-zero constraints, and uniform error spreading over the remaining K−1 classes (Eq. 8), fitted by penalized EM with a proved unique M-step maximizer (Proposition 4.2); (3) an evaluation on 33 real crowdsourcing datasets plus a synthetic benchmark, showing the highest average minority recall among six methods while remaining competitive in balanced accuracy, with the largest gains under strong imbalance, high annotation redundancy, and large-scale regimes. The theory, as sketched, is plausible: Theorem 3.1 reduces to a per-class application of a cited proposition, Corollary 3.2 is a total-variation argument, and Proposition 4.2 follows from strict concavity of the penalized Q-function plus coercivity. My concerns are mainly empirical-methodological (metric choice versus the paper's own per-dataset F1 numbers, and dataset accounting) and the dependence of Theorem 3.1 on an unpublished companion paper.

Significance. If the empirical claims hold up, the paper is a useful contribution: a principled generative aggregator for the practically important and under-studied imbalanced-crowdsourcing regime, with a uniqueness guarantee for the EM M-step, an unusually broad real-data evaluation (binary, multiclass, and two large-scale regimes), and a well-designed gold-supervision comparison in §5.3. The imbalanced Condorcet condition and the imbalance-preservation corollary are interpretable and extend prior work to heterogeneous multiclass crowds. The main risks are (i) that part of the minority-recall advantage is an operating-point artifact invisible to the chosen metrics, (ii) dataset-accounting errors that may inflate the benchmark's size and bias aggregates, and (iii) the key jury-theorem lemma living in an unpublished companion paper. All are addressable within the manuscript's scope.

major comments (5)
  1. [§5, Tables 1–4] The headline claim is 'highest minority recall while remaining competitive in balanced accuracy' (Abstract; Tables 1–3). Both metrics are recall-only: neither penalizes false positives on the majority class, so an aggregator that predicts the minority class more often inflates both. The paper's own Table 4 shows this signature: product (recall 0.825 vs DS 0.640 but Macro F1 0.707 vs 0.843), Loneliness Older Adults (F1 0.494 vs MV 0.667 at equal minority recall), Jigsaw (F1 0.918 vs DS 0.982), Amazon Sentiment Negative (F1 0.762 vs 0.90x for all baselines). Table 3, the only aggregate table with F1, has CC-Rasch below DS (0.812/0.793 vs 0.831/0.830). Per-dataset balanced accuracy is not reported, so the aggregate 'competitive' claim may hide per-dataset collapses. Please (a) report per-class precision or Macro F1 in aggregate Tables 1–2 alongside recall, (b) report per-dataset balanced ac
  2. [§5.2, Table 4] The '33 datasets' claim needs an audit. §5's text says 'we evaluate our CC-Rasch model across 27 real-world datasets' while the Abstract and Table 4 say 33. Table 4 contains 'bird' (Imb 1.28) and 'Bird' (Imb 1.25) as separate rows with otherwise identical statistics — apparently the same dataset counted twice. Weather Sentiment–AMT (Imb 10.22, 20.00 ann./item) has values numerically identical to CF* (10.22, 20.10) in every row, strongly suggesting a copy-paste error. CF and CF* share Imb 10.22. Since the benchmark's breadth is a stated contribution ('one of the largest benchmarks'), please provide a definitive dataset table (source, K, n, annotators, Imb), reconcile the 27/33 discrepancy, and correct or justify the duplicated rows, since duplicated near-balanced datasets would bias the aggregate means in Tables 1–2.
  3. [§3, Theorem 3.1] Theorem 3.1 is presented as a multiclass, multidimensional generalization, but the proof reduces entirely to 'Proposition D.3 of Singer et al. (2026) applies to each class k' — a lemma in the appendix of the authors' own unpublished preprint, presumably stated for the binary case. Two gaps: (i) For K > 2, the MV error event given Y = k is that some other class out-polls k; reducing this to a single binary 'good vs bad annotator' condition per class requires at minimum a union bound over the K−1 competing classes, which is not shown, and the pairwise margin needed depends on the runner-up class probabilities, not just p_r^(k). (ii) The condition uses the binary-flavored threshold p_r^(k) > 1/2 + δ_k, which is not the natural competence threshold for K > 2 (random guessing is 1/K). Please make the multiclass reduction explicit with the union-bound argument, and include a self-contained pro
  4. [§4.2, §5 (Implementation details)] The penalized EM has several tuning quantities — prior variances σ²_{α,k}, σ²_{β,k}, the mean penalties λ_{α,μ}, λ_{β,μ}, initialization of (G, H), number of EM restarts, and convergence criteria — none of which are reported in §5 (only 'L-BFGS-B with warm start' and uniform π init). Note also that Proposition 4.2 holds π fixed, so it is unclear whether/how π is updated, which matters under imbalance. Since the empirical comparison is the paper's central claim, please report these settings, a sensitivity analysis over the prior variances (the penalty both regularizes and, via coercivity, drives the uniqueness result), and state whether code will be released.
  5. [§4.2, Eq. (8)] Eq. (8) spreads annotation errors uniformly over the other K−1 classes, so the model captures class-dependent correctness but not class-dependent confusion structure. This is load-bearing for the model's motivation: if real annotators exhibit systematic majority→minority confusion (plausibly common in inspection tasks), the likelihood mis-weights exactly the minority specialists the model is designed to find. The synthetic benchmark (§5.1) is binary with symmetric error flips, so this failure mode is never exercised. Please add a synthetic multiclass experiment with an asymmetric confusion structure (e.g., a Dawid–Skene-style generator with majority→minority bias) showing whether CC-Rasch's recall advantage survives, and discuss the limitation explicitly.
minor comments (8)
  1. [Table 4] Table 4 names the proposed method 'Φ-Rasch (OUR)' while the rest of the paper calls it CC-Rasch (also '(OUR)' vs '(OURS)' across tables). Please unify.
  2. [§2–§4] Notation inconsistencies: §2 uses m annotators / n items; §3 uses R annotators with set RR; Proposition 4.1 says 'for any r ∈ [n]' (should be the annotator index set); §4.2 'for all (i,r) ∈ Ω_r × [R]' is malformed (Ω_r undefined; presumably Ω_i × ... or O); Corollary 3.2 uses max over 'k ∈ K' instead of [K].
  3. [§5.2, Table 2] PLAT is absent from the multiclass results (Table 2) without explanation; if it is binary-only, say so where baselines are introduced.
  4. [§4.1, Proposition 4.1] Proposition 4.1 (GLAD sensitivity = specificity) is stated without proof; a two-line derivation from Eq. (5) would suffice.
  5. [Throughout] Typos/formatting: 'on33real-world', 'a stronge class imbalance', 'one need to estimate', 'compare it performances', 'Wherem denotes', missing spaces in references ('PauliVirtanen'), 'École Polytechnique' citation style.
  6. [§1, Contributions] Contributions list says 'four baselines' but five are compared (MV, DS, GLAD, CrowdFM, PLAT).
  7. [§5.2, Table 4] In Table 4 most rows report ±0.000 across 4 seeds, suggesting deterministic fitting; please state which methods are seed-dependent and why variance vanishes, otherwise the '4 random seeds' protocol is unclear.
  8. [§5.1, Figure 1] Figure 1's dual x-axes (minority % decreasing left-to-right on top, majority % increasing on bottom) are easy to misread; consider plotting recall vs p_min only.

Circularity Check

1 steps flagged

No significant circularity: Condorcet result leans on one same-author lemma, but the model and minority-recall claims are estimated and scored against external ground truth.

specific steps
  1. self citation load bearing [§3 Theorem 3.1 proof]
    "Under condition (7), Proposition D.3 of Singer et al. (2026) applies to each class k∈[K], yielding lim R→∞ eR(k)=0. ... This is a generalization to the multidimension and multiclass case of Proposition D.3 in Singer et al. (2026)."

    The only non-elementary step in the imbalanced Condorcet argument is imported from a concurrent same-author paper rather than derived here. The present theorem is essentially 'apply our prior binary proposition class-wise under (7).' This is mild: condition (7) is stated independently, the reduction is explicit, and the central CC-Rasch/empirical claims do not depend on this lemma. Not a fit-as-prediction or definitional loop.

full rationale

The paper's load-bearing empirical claim (CC-Rasch highest minority recall on 33 datasets) is not circular: parameters are fit by penalized EM on observed crowd labels only, and minority recall / balanced accuracy are computed against held-out ground-truth labels that never enter the likelihood. The generative model (class-conditional logistic correctness, uniform error spread, Gaussian mixed effects, sum-to-zero constraints) is an ansatz, not a quantity defined from the evaluation metrics. Proposition 4.1 (GLAD sensitivity=specificity) and Proposition 4.2 (unique M-step maximizer via strict concavity + coercivity on a closed convex set) are self-contained derivations from the stated equations. Theorem 3.1 and Corollary 3.2 state crowd-composition hypotheses and deduce asymptotic consistency and proportion preservation; they are not fitted to the tables. The only mild circularity-adjacent step is that the multiclass Condorcet proof invokes Proposition D.3 of Singer et al. (2026)—same author list—rather than re-proving the binary building block in-line. That citation is not load-bearing for the model or the empirical minority-recall claim, and does not force the reported metrics by construction. Score 1.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The central empirical claim rests on the CC-Rasch likelihood and EM fitting; the theory rests on conditional independence of annotators and competence lower bounds. Free parameters are the usual IRT/EM variances and class proportions. No exotic physical entities; invented pieces are modeling constructs standard in psychometrics/crowdsourcing.

free parameters (3)
  • Prior variances σ²_{α,k}, σ²_{β,k} (and optional μ penalties)
    Gaussian mixed-effect priors fix scale/translation; values are not reported as data-driven fits in the main text but control regularization strength of g_{r,k}, h_{i,k}.
  • Class proportions π_k = uniform initialization
    Initialized uniformly and used in the complete-data likelihood / E-step; can be updated or held as part of the generative model.
  • Synthetic reliability table q_{t,k}, hard-item penalty δ, imbalance π, type proportions = p_good=p_bad=25%, p_maj+p_min=50%, Imb≈7
    Hand-chosen generative parameters for the controlled specialist sweep in §5.1 / Fig. 1.
axioms (5)
  • domain assumption Annotator labels are conditionally independent given the true item label Y.
    Stated explicitly before Theorem 3.1; standard crowdsourcing assumption used for both jury theorem and likelihood factorization.
  • ad hoc to paper P(correct | Y=k) = σ(α_{r,k} − β_{i,k}) with errors uniform on the remaining classes (Eq. 8).
    Core CC-Rasch observation model; extends Rasch/GLAD but the uniform multiclass error spread is a modeling choice not forced by data.
  • ad hoc to paper Sum-to-zero constraints ∑_r g_{r,k}=∑_i h_{i,k}=0 and independent Gaussian priors on deviations.
    Identifiability / mixed-effects regularization device in §4.2, following Whitehill/Liu-style practice.
  • domain assumption For Condorcet: liminf crowd composition inequality (7) with class-wise specialist sets G^{(k)} and bad sets B^{(k)}.
    Hypothesis of Theorem 3.1; if violated (too few minority specialists), consistency need not hold.
  • standard math Logistic link and Bernoulli correctness indicators as in classical Rasch/GLAD.
    Inherited item-response machinery (Rasch 1960; Whitehill et al. 2009).
invented entities (2)
  • CC-Rasch (class-conditional mixed-effects Rasch) parameter set (μ_α, μ_β, G, H) no independent evidence
    purpose: Jointly represent per-class annotator ability and per-class item difficulty for latent-label EM aggregation.
    New model class relative to single-ability GLAD and confusion-matrix Dawid–Skene; still a parametric statistical model, not a new physical object.
  • Minority-class / majority-class specialist partitions G^{(j_min)}, G^{(j_max)} independent evidence
    purpose: Decompose the crowd for imbalanced Condorcet analysis and synthetic robustness sweeps.
    Definitional subsets based on p^{(k)}_r thresholds; useful taxonomy rather than a latent discovered substance.

pith-pipeline@v1.2.0-grok45-kimik3 · 23476 in / 3741 out tokens · 58142 ms · 2026-07-31T10:17:37.093274+00:00 · methodology

0 comments
read the original abstract

We study imbalanced crowdsourcing with a focus on class-dependent annotator accuracy, a setting that, to the best of our knowledge, remains relatively underexplored despite its importance in real-world inspection systems where the labels of greatest operational importance are also the rarest ones. In this setting, annotators may be reliable on both classes, unreliable on both classes, majority-class specialists, or minority-class specialists. Existing models only partially address this problem: they either capture class-dependent errors but ignore item difficulty, or they model item difficulty without capturing class-dependent errors. To fill this gap for imbalanced datasets in crowdsourcing, we introduce a generative aggregation model combining item difficulty with class-dependent annotator competence. The model allows both annotator abilities and item difficulties to vary across classes. We then revisit Condorcet's Jury Theorem in the class-imbalanced setting. We also show that majority voting asymptotically preserves the underlying class proportion. We evaluate our model on $33$ real-world crowdsourcing datasets, covering multiclass tasks such as images and text, as well as two large-scale regimes: large-scale annotation datasets, with many annotations per item, and large-scale item datasets, with a large number of annotated instances. Across these diverse settings, our model consistently achieves the highest minority recall while remaining competitive in balanced accuracy, making it particularly relevant when rare-label recovery is the primary objective.

Figures

Figures reproduced from arXiv: 2607.24622 by Argyris Kalogeratos, Gabriel Singer, Nicolas Vayatis, Olivier Vo Van, Samuel Gruffaz.

Figure 1
Figure 1. Figure 1: Minority-class recall as the proportion of majority-class specialists [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Gold label boosted methods versus CC-Rasch. [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

159 extracted references · 4 canonical work pages

  1. [1]

    , title =

    Whitehill, Jacob and Ruvolo, Paul and Wu, Ting-fan and Bergsma, Jacob and Movellan, Javier R. , title =. Advances in Neural Information Processing Systems , volume =

  2. [2]

    Intelligent Data Analysis , volume =

    Japkowicz, Nathalie and Stephen, Shaju , title =. Intelligent Data Analysis , volume =. 2002 , doi =

    2002

  3. [3]

    and Jordan, Michael I

    Wainwright, Martin J. and Jordan, Michael I. , title =. Foundations and Trends in Machine Learning , volume =. 2008 , doi =

    2008

  4. [4]

    , title =

    Nguyen, Duc and Zhang, Anderson Y. , title =. Advances in Neural Information Processing Systems , volume =

  5. [5]

    Rasch, Georg , title =

  6. [6]

    and Laird, Nan M

    Dempster, Arthur P. and Laird, Nan M. and Rubin, Donald B. , title =. Journal of the Royal Statistical Society: Series B (Methodological) , volume =. 1977 , doi =

    1977

  7. [7]

    and Steinhardt, Jacob , title =

    Lipton, Zachary C. and Steinhardt, Jacob , title =. Queue , volume =. 2019 , publisher =

    2019

  8. [8]

    and Chiang, Mung and Studer, Christoph , title =

    Lan, Andrew S. and Chiang, Mung and Studer, Christoph , title =. Proceedings of the 35th International Conference on Machine Learning , series =

  9. [9]

    and Plan, Yaniv and van den Berg, Ewout and Wootters, Mary , title =

    Davenport, Mark A. and Plan, Yaniv and van den Berg, Ewout and Wootters, Mary , title =. Information and Inference: A Journal of the IMA , volume =. 2014 , doi =

    2014

  10. [10]

    , title =

    Andersen, Erling B. , title =. Journal of the Royal Statistical Society: Series B (Methodological) , volume =. 1970 , doi =

    1970

  11. [11]

    Statistics & Probability Letters , volume =

    Ghosh, Malay , title =. Statistics & Probability Letters , volume =. 1995 , doi =

    1995

  12. [12]

    Haberman , title =

    Shelby J. Haberman , title =. The Annals of Statistics , number =. 1977 , doi =

    1977

  13. [13]

    Wainwright and Bin Yu , title =

    Sivaraman Balakrishnan and Martin J. Wainwright and Bin Yu , title =. The Annals of Statistics , number =. 2017 , doi =

    2017

  14. [14]

    2019 , eprint=

    Class-Balanced Loss Based on Effective Number of Samples , author=. 2019 , eprint=

    2019

  15. [15]

    2026 , eprint=

    MIFair: A Mutual-Information Framework for Intersectionality and Multiclass Fairness , author=. 2026 , eprint=

    2026

  16. [16]

    1997 , publisher=

    Spectral Graph Theory , author=. 1997 , publisher=

    1997

  17. [17]

    2023 , eprint=

    Fairness Guarantee in Multi-Class Classification , author=. 2023 , eprint=

    2023

  18. [18]

    International Conference on Machine Learning , pages=

    Fair and optimal classification via post-processing , author=. International Conference on Machine Learning , pages=

  19. [19]

    Preprint arXiv:2202.09724 , year=

    Bayes-optimal classifiers under group fairness , author=. Preprint arXiv:2202.09724 , year=

    Pith/arXiv arXiv

  20. [20]

    2022 , eprint=

    Blackbox Post-Processing for Multiclass Fairness , author=. 2022 , eprint=

    2022

  21. [21]

    Proceedings of 2nd Workshop on Diversity in Artificial Intelligence (AIDBEI) , year =

    Characterizing Intersectional Group Fairness with Worst-Case Comparisons , author =. Proceedings of 2nd Workshop on Diversity in Artificial Intelligence (AIDBEI) , year =

  22. [22]

    Shuman, D. I. and Narang, S. K. and Frossard, P. and Ortega, A. and Vandergheynst, P. , year=. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains , volume=. IEEE Signal Processing Magazine , publisher=. doi:10.1109/msp.2012.2235192 , number=

    arXiv 2012

  23. [23]

    Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining , pages=

    Algorithmic decision making and the cost of fairness , author=. Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining , pages=

  24. [24]

    Fairness definitions explained , doi =

    Verma, Sahil and Rubin, Julia , year =. Fairness definitions explained , doi =

  25. [25]

    2018 , eprint=

    Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness , author=. 2018 , eprint=

    2018

  26. [26]

    2025 , eprint=

    Beyond Flatland: A Geometric Take on Matching Methods for Treatment Effect Estimation , author=. 2025 , eprint=

    2025

  27. [27]

    2011 , eprint=

    Fairness Through Awareness , author=. 2011 , eprint=

    2011

  28. [28]

    The distance function and Lipschitz classes of mappings between metric spaces , volume =

    Markovic, Marijan , year =. The distance function and Lipschitz classes of mappings between metric spaces , volume =. Mathematika , doi =

  29. [29]

    CoRR , volume =

    Moritz Hardt and Eric Price and Nathan Srebro , title =. CoRR , volume =. 2016 , url =. 1610.02413 , timestamp =

    Pith/arXiv arXiv 2016

  30. [30]

    Algorithmic decision making and the cost of fairness , journal =

    Sam Corbett. Algorithmic decision making and the cost of fairness , journal =. 2017 , url =. 1701.08230 , timestamp =

    Pith/arXiv arXiv 2017

  31. [31]

    Journal of Machine Learning Research , year =

    Hilde Weerts and Miroslav DudÃk and Richard Edgar and Adrin Jalali and Roman Lutz and Michael Madaio , title =. Journal of Machine Learning Research , year =

  32. [32]

    The Risk of Racial Bias in Hate Speech Detection

    Sap, Maarten and Card, Dallas and Saeidi, Saeid and Kleinberg, Adam and Jurafsky, Dan and Smith, Noah A. The Risk of Racial Bias in Hate Speech Detection. 57th Annual Meeting of the Association for Computational Linguistics. 2019

    2019

  33. [33]

    Consistency of weighted majority votes , volume =

    Berend, Daniel and Kontorovich, Aryeh , booktitle =. Consistency of weighted majority votes , volume =

  34. [34]

    Advances in Neural Information Processing Systems , volume=

    The multidimensional wisdom of crowds , author=. Advances in Neural Information Processing Systems , volume=

  35. [35]

    Advances in Neural Information Processing Systems , volume=

    Whose vote should count more: Optimal integration of labels from labelers of unknown expertise , author=. Advances in Neural Information Processing Systems , volume=

  36. [36]

    World Wide Web Conference , pages=

    Nuanced metrics for measuring unintended bias with real data for text classification , author=. World Wide Web Conference , pages=

  37. [37]

    International Economic Review , volume=

    Optimal decision rules in uncertain dichotomous choice situations , author=. International Economic Review , volume=. 1982 , publisher=

    1982

  38. [38]

    Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing , pages=

    Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets , author=. Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing , pages=

  39. [39]

    Preprint arXiv:2110.04052 , year=

    On Release! 50 Annotator Labels of 10K Toxicity Data , author=. Preprint arXiv:2110.04052 , year=

    Pith/arXiv arXiv

  40. [40]

    International Conference on Machine Learning , pages =

    A reductions approach to fair classification , author =. International Conference on Machine Learning , pages =

  41. [41]

    Advances in Neural Information Processing Systems , volume =

    Empirical risk minimization under fairness constraints , author =. Advances in Neural Information Processing Systems , volume =

  42. [42]

    2019 , eprint=

    Human uncertainty makes classification more robust , author=. 2019 , eprint=

    2019

  43. [43]

    Learning with Noisy Labels Revisited: A Study on

    Wei, Jiaheng and Zhu, Zhaowei and Cheng, Hao and Liu, Tongliang and Niu, Gang and Liu, Yang , booktitle=. Learning with Noisy Labels Revisited: A Study on

  44. [44]

    Maximum Likelihood Estimation of Observer Error-Rates Using the

    Dawid, Alexander Philip and Skene, Allan M , journal=. Maximum Likelihood Estimation of Observer Error-Rates Using the. 1979 , publisher=

    1979

  45. [45]

    Civil Engineering and Environmental Systems , volume=

    Evaluation of inspection features including exposure risk using a value of information analysis , author=. Civil Engineering and Environmental Systems , volume=. 2021 , publisher=

    2021

  46. [46]

    Preprint arXiv:2210.06812 , year =

    CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators , author =. Preprint arXiv:2210.06812 , year =

    Pith/arXiv arXiv

  47. [47]

    ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages=

    Towards fair truth discovery from biased crowdsourced answers , author=. ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages=

  48. [48]

    Test , volume=

    Reducing degradation and age of items in imperfect repair modeling , author=. Test , volume=. 2022 , publisher=

    2022

  49. [50]

    Advances in Neural Information Processing Systems , volume =

    Counterfactual fairness , author =. Advances in Neural Information Processing Systems , volume =

  50. [51]

    2023 , school =

    Uncovering Bias and Fairness in Crowdsourced Labeling , author =. 2023 , school =

    2023

  51. [52]

    International Conference on Machine Learning , pages =

    Fairness-aware learning for continuous attributes and treatments , author =. International Conference on Machine Learning , pages =

  52. [53]

    2018 , publisher=

    Lectures on the Poisson process , author=. 2018 , publisher=

    2018

  53. [54]

    International Conference on Machine Learning , pages=

    Gaussian process classification and active learning with multiple annotators , author=. International Conference on Machine Learning , pages=

  54. [55]

    2019 , publisher=

    Multilevel modeling , author=. 2019 , publisher=

    2019

  55. [56]

    IEEE Access , volume=

    Mixed-effects nonhomogeneous Poisson process model for multiple repairable systems , author=. IEEE Access , volume=. 2021 , publisher=

    2021

  56. [57]

    , author =

    Learning from crowds. , author =. Journal of Machine Learning Research , volume =

  57. [58]

    Jigsaw unintended bias in toxicity classification , howpublished =

  58. [59]

    Knowledge and Information Systems , volume=

    Data preprocessing techniques for classification without discrimination , author=. Knowledge and Information Systems , volume=. 2012 , publisher=

    2012

  59. [60]

    Science advances , volume=

    The accuracy, fairness, and limits of predicting recidivism , author=. Science advances , volume=. 2018 , publisher=

    2018

  60. [61]

    AAAI/ACM Conference on AI, Ethics, and Society , pages=

    Crowdsourcing with fairness, diversity and budget constraints , author=. AAAI/ACM Conference on AI, Ethics, and Society , pages=

  61. [62]

    Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages =

    Bayesian crowdsourcing with constraints , author =. Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages =. 2021 , organization =

    2021

  62. [63]

    Preprint arXiv:2005.00582 , year =

    Learning to complement humans , author =. Preprint arXiv:2005.00582 , year =

    Pith/arXiv arXiv 2005

  63. [64]

    Conference on Causal Learning and Reasoning , pages =

    Fair classification with instance-dependent label noise , author =. Conference on Causal Learning and Reasoning , pages =

  64. [65]

    Pattern Recognition , pages =

    Learning to Complement with Multiple Humans , author =. Pattern Recognition , pages =. 2025 , publisher =

    2025

  65. [66]

    2021 , eprint=

    Binary classification with corrupted labels , author=. 2021 , eprint=

    2021

  66. [67]

    Classification in the Presence of Label Noise: A Survey , year=

    Frenay, Benoit and Verleysen, Michel , journal=. Classification in the Presence of Label Noise: A Survey , year=

  67. [68]

    ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

    To aggregate or not? learning with separate noisy labels , author=. ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

  68. [69]

    and Cogswell, Michael and Das, Abhishek and Vedantam, Ramakrishna and Parikh, Devi and Batra, Dhruv , year=

    Selvaraju, Ramprasaath R. and Cogswell, Michael and Das, Abhishek and Vedantam, Ramakrishna and Parikh, Devi and Batra, Dhruv , year=. International Journal of Computer Vision , publisher=

  69. [70]

    Raykar and Shipeng Yu and Linda H

    Vikas C. Raykar and Shipeng Yu and Linda H. Zhao and Gerardo Hermosillo Valadez and Charles Florin and Luca Bogoni and Linda Moy , title =. Journal of Machine Learning Research , year =

  70. [71]

    5th International Conference on Learning Representations , year =

    Jacob Goldberger and Ehud Ben-Reuven , title =. 5th International Conference on Learning Representations , year =

  71. [72]

    Using Boosting to Detect Noisy Data

    Wheway, Virginia. Using Boosting to Detect Noisy Data. Advances in Artificial Intelligence. PRICAI 2000 Workshop Reader. 2001

    2000

  72. [73]

    Machine Learning , year=

    Angluin, Dana and Laird, Philip , title=. Machine Learning , year=

  73. [74]

    2022 , eprint=

    Learning from Noisy Labels with Deep Neural Networks: A Survey , author=. 2022 , eprint=

    2022

  74. [75]

    Advances in Neural Information Processing Systems 30 (NeurIPS 2017) , pages =

    Eran Malach and Shai Shalev-Shwartz , title =. Advances in Neural Information Processing Systems 30 (NeurIPS 2017) , pages =

    2017

  75. [76]

    MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels , booktitle =

    Lu Jiang and Zhengyuan Zhou and Thomas Leung and Li. MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels , booktitle =

  76. [77]

    Tsang and Masashi Sugiyama , title =

    Bo Han and Quanming Yao and Xingrui Yu and Gang Niu and Miao Xu and Weihua Hu and Ivor W. Tsang and Masashi Sugiyama , title =. Advances in Neural Information Processing Systems (NeurIPS) , pages =

  77. [78]

    Aritra Ghosh and Naresh Manwani and P. S. Sastry , title =. AAAI Conference on Artificial Intelligence , pages =

  78. [79]

    Advances in Neural Information Processing Systems (NeurIPS) , pages =

    Zhilu Zhang and Murat Sabuncu , title =. Advances in Neural Information Processing Systems (NeurIPS) , pages =

  79. [80]

    Conference on Empirical Methods in Natural Language Processing , pages=

    Cheap and fast--but is it good? evaluating non-expert annotations for natural language tasks , author=. Conference on Empirical Methods in Natural Language Processing , pages=

  80. [81]

    IEEE International Conference on Computer Vision , pages =

    Yisen Wang and Xingjun Ma and Zaiyi Chen and Yuan Luo and James Bailey and Jinfeng Yi , title =. IEEE International Conference on Computer Vision , pages =

Showing first 80 references.