Pith. sign in

REVIEW 4 major objections 5 minor 48 references

A Framework for Evaluating LLMs Under Task Indeterminacy

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Scoring LLMs against a single gold label understates their true accuracy whenever tasks have more than one valid answer; the paper defines the right target as membership in the valid response set and supplies bounds to correct the bias.

desk verdict Clean definitional point about gold-label underestimation under task indeterminacy, but the proposed bounds are unverified heuristics and the paper needs real-data validation before the practical method is credible. read the letter →

arxiv 2411.13760 v1 pith:AADCJBXX submitted 2024-11-21 cs.LG cs.CLcs.HC

Luke Guerdan , Hanna Wallach , Solon Barocas , Alexandra Chouldechova This is my paper · ORCID
classification cs.LGcs.CLcs.HC
keywords taskindeterminacyvalidresponsesetLLMevaluationgoldlabelbiashumanratingvariationambiguityvaguenessperformancebounds
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM benchmarks usually score a model by counting how often its answer matches one pre-chosen gold label, but many tasks have no unique correct answer. This paper argues that the right target is whether the model's response lands in the valid response set, meaning the set of responses correct under at least one reasonable reading of the instruction, and defines true performance as $M^*(Y_L, Y_v) = P(Y_L \in Y_v)$. A synthetic experiment consistent with the paper's causal model shows that the standard gold-label metric systematically underestimates this true performance, with the gap growing as the proportion of indeterminate items increases. The paper offers two ways to bound the unmeasured true score from partial knowledge: a prevalence bound based on the share of indeterminate items, and a tighter partition bound that splits the corpus into determinate and indeterminate items by human-rater agreement.

What carries the argument

The load-bearing objects are the valid response set (VRS), defined as the set of responses correct under at least one reasonable interpretation of the instruction text, and the two performance measures $M(Y_L, Y_R) = P(Y_L = Y_R)$ versus $M^*(Y_L, Y_v) = P(Y_L \in Y_v)$. The argument is carried by a causal directed acyclic graph of the evaluation pipeline that separates instruction text, rater identity, interpretation, rater error, and LLM response, so that indeterminacy-driven variation is distinguished from exogenous noise. On top of this, the prevalence bound and the partition bound convert partial knowledge about indeterminate items into an interval estimate of true performance, with the partition bound using an agreement-based split of the corpus into determinate and indeterminate subsets.

What would settle it

On a forced-choice benchmark, elicit the valid response set for every item from a diverse panel using instructions that explicitly separate 'multiple reasonable answers' from 'rater mistake,' then compute both the gold-label score and the VRS score. If the gold-label score is not strictly below the VRS score on a corpus with a substantial measured share of indeterminate items, or if the partition bound fails to contain the VRS score, the claimed bias direction and the bound procedure would both be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that task indeterminacy, meaning ambiguity or vagueness that leaves more than one correct response, is not a nuisance to be averaged away but a measurable feature of the evaluation corpus, and that ignoring it biases LLM performance estimates downward. Formally, the correct performance measure is $M^*(Y_L, Y_v) = P(Y_L \in Y_v)$, the probability that the LLM response belongs to the item's valid response set; the conventional measure $M(Y_L, Y_R) = P(Y_L = Y_R)$, based on a single aggregated gold label, counts legitimate alternative responses as errors. Since the gold label is itself one of the valid responses whenever aggregation picks a reasonable answer, the event $\{Y_L = Y_R\}$ is contained in $\{Y_L \in Y_v\}$, so the gold-label score cannot exceed the true score, and the shortfall grows with the fraction of indeterminate items, as shown in the synthetic DAG-consistent experiment. The paper further proposes estimating this unmeasured quantity with an interval: the prevalence bound uses only the estimated proportion of indeterminate items, while the partition bound additionally labels individual items as determinate versus indeterminate, for example by an agreement threshold, and brackets true performance more tightly.

Load-bearing premise

The method assumes that low human-rater agreement reliably marks an item as having multiple valid answers, rather than being a sign of sloppy or error-prone raters; if agreement cannot be separated from exogenous error, the bounds will misclassify items and may fail to bracket the true score.

Editorial extensions

If this is right

  • Every reported gold-label accuracy number on a corpus containing ambiguous or vague items is a lower bound on the model's true accuracy under the paper's definition, so benchmark comparisons should be read as conservative.
  • Evaluation designers can deploy the prevalence bound with only a small random audit of items, and the tighter partition bound by labeling items via an agreement threshold, without enumerating every valid response set.
  • The DAG gives a shared language for where subjectivity enters evaluation, letting designers attribute score changes to instruction text, rater identity, or model behavior rather than to a single aggregate number.
  • Improvements to task instructions or annotation guidelines can be quantified as reductions in the gap between the gold-label estimate and the bounded true-performance interval.
  • Models that choose a reasonable but non-majority response on an indeterminate item are counted as wrong by gold-label scoring and would be re-scored as correct under VRS-based evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the bias is as systematic as argued, model rankings can shift: a model that frequently produces reasonable-but-non-majority answers is under-rewarded by gold-label scoring, so leaderboard ranks are not stable under a VRS re-scoring.
  • A natural testable extension is to reuse existing disagreement-heavy datasets, such as safety or toxicity annotation with many raters, and check whether the partition bound, using human agreement as the split rule, brackets a VRS score measured by an independent elicitation procedure.
  • The framework suggests a design experiment: rewriting instructions to reduce ambiguity should shrink the gap between $M$ and $M^*$, giving evaluation designers a steerable, quantitative target for prompt and guideline refinement.
  • The same DAG logic applies beyond forced-choice items, which the paper explicitly leaves open; adapting the VRS to open-ended outputs would require defining a membership test for generated text, a harder but well-posed problem.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that standard LLM evaluations which assume a single gold label per item are biased when the corpus contains indeterminate items, i.e., items with more than one valid response. It formalizes this with a causal DAG that distinguishes task specification, human rating processes, evaluator unobservables, and LLM responses, and it defines 'true performance' as membership in a valid response set (VRS). The authors present a synthetic experiment (Figure 2) claiming to show that gold-label evaluations underestimate true performance and that the bias grows with the proportion of indeterminate items. They also propose two methods for bounding true performance from partial knowledge: a prevalence bound and a partition bound based on human-rater agreement thresholds. The paper concludes by discussing implications for evaluation design and acknowledges limitations including lack of case studies and incomplete treatment of evaluation reliability and validity.

Significance. If the framework is developed rigorously, it addresses a real and timely gap: quantifying how task ambiguity and vagueness distort LLM performance estimates. The DAG-based disentanglement of task specification, rater error, and LLM behavior is a useful conceptual contribution, and the definitional point that a single gold label can understate accuracy when multiple responses are valid is correct. However, the current manuscript is a short position/workshop-style paper: the central underestimation claim is largely built into the definitions, the synthetic experiment is not reproducible from the text, the proposed bounds have no derivation or coverage guarantees, and the limitations section explicitly disclaims the empirical and methodological completeness needed for a journal-level contribution. The significance is therefore conditional on substantial further development and validation.

major comments (4)
  1. [§4, definition of true performance] The claim that gold-label evaluation underestimates true performance is not established as an empirical finding; it follows directly from the definitions once one assumes that the gold label Y_R is an element of the VRS Y_v for each item. The paper never states this membership assumption or proves the inequality M(Y_L, Y_R) ≤ M*(Y_L, Y_v). As written, the 'underestimation' direction is a definitional artifact rather than a substantive result. Please state the formal conditions under which Y_R ∈ Y_v (e.g., that the rating-aggregation procedure always returns one valid response) and present the inequality as a short proposition. This would clarify what the synthetic experiment in Figure 2 actually adds beyond the definitions.
  2. [§4, Figure 2 and synthetic experiment] The synthetic experiment is not described in sufficient detail to assess any quantitative claim. The text says only that data were 'randomly generated consistent with our DAG'; it does not report the generative model, the number of items, the number of raters, the threshold used for the partition bound, the prevalence values, or the number of simulated trials. Figure 2 shows curves for prevalence bound, partition bound, true performance, and gold-label assumption, but there are no error bars or repeated-run variability. Consequently, the monotonic increase in bias and the relative widths of the bounds cannot be verified or reproduced. Please provide a complete generative specification (possibly in an appendix), report parameter choices and sample sizes, and include confidence intervals or repeated seeds.
  3. [§4, partition bound] The partition bound is presented only as a heuristic: sort items by human-rater (or LLM) agreement and select a threshold below which items are deemed indeterminate. No derivation shows that the resulting interval brackets M*(Y_L, Y_v), and no conditions are given on rater error rates, agreement distributions, or threshold selection. Because the paper's own DAG separates Rater Unobservables and Evaluator Unobservables from the latent intent that determines the VRS, low agreement is equally consistent with exogenous rater error as with genuine indeterminacy, and high agreement on an indeterminate item cannot be excluded. As a result, the bound has no stated coverage guarantee and may be neither an upper nor a lower bound in realistic settings. Please either provide formal conditions under which the interval contains M*(Y_L, Y_v), or reframe the proposal as an exploratory diagnostic rather than an 'error-adjusted performance interval.'
  4. [Limitations] The Limitations section explicitly states that the framework 'does not offer a comprehensive assessment of evaluation reliability and validity,' that the authors were 'unable to provide worked case studies or framework applications,' and that the DAG 'may omit factors.' These admissions are directly relevant to the paper's central practical claim that it provides 'a method for estimating an error-adjusted performance interval.' Without case studies, a fully specified synthetic study, or a real-data application, the utility of the method for evaluation designers is not demonstrated. These limitations should be moved into the body as scoping conditions and addressed with at least one concrete application or validation before the manuscript can support its conclusions.
minor comments (5)
  1. [§4, definition of indeterminate items] The text says indeterminate items are those with '|VRS| ≥ 1,' but under the definition given, determinate items have |VRS| = 1, so indeterminate items should be defined by |VRS| ≥ 2. This appears to be a typo, but it makes the partition definition incoherent and should be corrected.
  2. [References] There are duplicate references for the same works: [8] and [9] both cite Chen and Zhang, 'Judgment Sieve'; and [18] and [19] both cite Gordon et al., 'The Disagreement Deconvolution.' Please consolidate these entries.
  3. [Figure 1] Figure 1 is described only as 'an overview of our causal DAG'; the right panel expansion and the red-highlighted path from Human Rating to Performance Metric to denote the gold-label approach are never explained in the text. Please add a walkthrough of the key paths and variables.
  4. [§3 and Figure 2 caption] The paper uses both Y_v and Yv for the valid response set, and it is not always clear whether Y_R denotes the aggregate single gold label or the raw rating of an individual rater. Please standardize notation and define all symbols at first use.
  5. [Limitations] The first sentence of the Limitations section, 'One limitation of our framework (Figure 2) is that its scope is limited to forced-choice NLP tasks,' is confusing: Figure 2 is an empirical plot, not the framework. Please replace '(Figure 2)' with a reference to the relevant section or definition, or remove it.

Circularity Check

1 steps flagged · score 3.0 of 10

The underestimation result is a definitional consequence of defining true performance as membership in the valid response set, not an independent empirical finding.

  1. self definitional [Section 4 (definitions of M and M*; Figure 2).]
    "However, when tasks are indeterminate, an LLM response to an item should be deemed correct if it matches any of the responses in that item's VRS. We therefore define the true performance as M ∗( ˆYL, Yv) = P( ˆYL ∈ Yv). ... Figure 2 shows that evaluations that use the gold label assumption underestimate the true performance. ... Intuitively, this is because task indeterminacy introduces additional ways for the LLM's responses to be correct."

    The inequality is forced by the paper's own definitions. The gold-label metric M(Y_L, Y_R)=P(Y_L=Y_R) counts only responses equal to the aggregated rating, whereas true performance M*(Y_L, Y_v)=P(Y_L in Y_v) counts responses landing anywhere in the VRS. Because the paper treats the gold label as a singleton approximation of the VRS, the event {Y_L=Y_R} is a subset of {Y_L in Y_v} whenever the approximation is faithful, making M <= M* a set-theoretic consequence rather than an empirical discovery. The authors' explanation that indeterminacy 'introduces additional ways for the LLM's responses to be correct' confirms the direction of the inequality is built into the definition of true performance.

full rationale

The paper contains no load-bearing self-citations and no fitted-parameter-then-prediction chain. The prevalence and partition bounds are presented as heuristics with no closed-form derivation, so no circular reduction can be exhibited for them; their validity is a correctness and coverage concern rather than a circularity concern. The one genuine circular step is the central 'underestimation' result: once true performance is defined as membership in the valid response set and the gold label is described as a singleton approximation of that set, the inequality M(Y_L,Y_R) <= M*(Y_L,Y_v) follows by construction. This makes the headline empirical claim partly a definitional artifact. However, the paper's framework, DAG, and discussion of ambiguity versus rater error retain independent conceptual content, so the overall circularity is modest rather than pervasive.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The framework's central claims depend on the existence and estimability of the VRS and on the assumed DAG structure. The prevalence bound requires an estimated proportion of indeterminate items, and the partition bound requires a chosen agreement threshold; both are free quantities. The synthetic experiment's generative parameters are unreported, so the plotted results cannot be reproduced.

free parameters (3)
  • Proportion of indeterminate items (prevalence)
    The prevalence bound requires an estimate of the fraction of items with |VRS| > 1; this is a free parameter estimated from a sample.
  • Agreement threshold for partition bound
    The partition bound sorts items by human-rater agreement and selects a threshold below which items are deemed indeterminate; the threshold is a heuristic choice not specified.
  • Synthetic experiment generative parameters
    The synthetic experiment uses 'randomly generated data consistent with our DAG' but no parameters (e.g., rater error rates, model accuracy on determinate vs indeterminate items) are reported, so the curves are effectively produced by an unspecified fitted model.
assumptions (3)
  • domain assumption The valid response set (VRS) is well-defined and contains exactly the responses correct for at least one reasonable interpretation of the instruction text.
    Invoked in Section 3 when defining Y_v and in the definition of true performance M* in Section 4. The paper provides no procedure or evidence that such sets can be reliably elicited.
  • domain assumption The DAG structure correctly represents the causal relationships in the LLM evaluation pipeline (e.g., interpretation, rater identity, instruction text affect human ratings).
    The entire framework and the synthetic experiment assume this causal graph (Figure 1, Section 3). No empirical validation is given, and the Limitations section admits the DAG may omit important factors.
  • ad hoc to paper Human rater agreement is a valid proxy for distinguishing determinate from indeterminate items.
    The partition bound heuristic sorts items by agreement and thresholds it, but disagreement could reflect exogenous error rather than indeterminacy (a distinction the paper itself draws). This assumption is introduced without justification in Section 4.
invented entities (1)
  • Valid response set (VRS)
    purpose: Defines the set of all responses correct for at least one reasonable interpretation, used to define true performance M*.
    The paper suggests the VRS can be estimated via crowdsourcing, but provides no data, protocol, or measurement. It is a theoretical construct with no falsifiable handle established in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Framework for Evaluating LLMs Under Task Indeterminacy." pith.science (2026). https://pith.science/paper/AADCJBXX

@misc{pith2026241113760,
  author       = {Pith},
  title        = {Pith review of: A Framework for Evaluating LLMs Under Task Indeterminacy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AADCJBXX}},
  note         = {Machine review of arXiv:2411.13760}
}
read the original abstract

Large language model (LLM) evaluations often assume there is a single correct response -- a gold label -- for each item in the evaluation corpus. However, some tasks can be ambiguous -- i.e., they provide insufficient information to identify a unique interpretation -- or vague -- i.e., they do not clearly indicate where to draw the line when making a determination. Both ambiguity and vagueness can cause task indeterminacy -- the condition where some items in the evaluation corpus have more than one correct response. In this paper, we develop a framework for evaluating LLMs under task indeterminacy. Our framework disentangles the relationships between task specification, human ratings, and LLM responses in the LLM evaluation pipeline. Using our framework, we conduct a synthetic experiment showing that evaluations that use the "gold label" assumption underestimate the true performance. We also provide a method for estimating an error-adjusted performance interval given partial knowledge about indeterminate items in the evaluation corpus. We conclude by outlining implications of our work for the research community.

Figures

Figures reproduced from arXiv: 2411.13760 by the authors.

Figure 1
Figure 1. An overview of our causal directed acyclic graph [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 22 canonical work pages

  1. [1]

    DICES Dataset: Diversity in Conversational AI Evaluation for Safety

    Lora Aroyo, Alex S Taylor, Mark D ´ıaz, Christopher M Homan, Alicia Parrish, Greg Serapio-Garc ´ıa, Vinodkumar Prabhakaran, and Ding Wang. DICES Dataset: Diversity in Conversational AI Evaluation for Safety

    work page

  2. [2]

    Stop Measuring Calibration When Humans Disagree, November 2022

    Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fern´andez. Stop Measuring Calibration When Humans Disagree, November 2022. URL http://arxiv.org/abs/2210.16133. arXiv:2210.16133 [cs]

    arXiv 2022

  3. [3]

    It’s the End of the Gold Standard as we Know it

    Valerio Basile. It’s the End of the Gold Standard as we Know it. On the Impact of Pre-aggregation on the Evaluation of Highly Subjective Tasks

    work page

  4. [4]

    Like trainer, like bot? Inheritance of bias in algorithmic content moderation

    Reuben Binns, Michael Veale, Max Van Kleek, and Nigel Shadbolt. Like trainer, like bot? In- heritance of bias in algorithmic content moderation. volume 10540, pages 405–415. 2017. doi: 10.1007/978-3-319-67256-4 32. URL http://arxiv.org/abs/1707.01477. arXiv:1707.01477 [cs]. 3For example, consider an annotation guideline that instructs a rater to label a c...

    work page Pith review arXiv 2017

  5. [5]

    Holistic evaluation of language models

    Rishi Bommasani, Percy Liang, and Tony Lee. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525(1):140–146, 2023

    work page 2023

  6. [6]

    Jonathan Bragg, Mausam, and Daniel S. Weld. Sprout: Crowd-Powered Task Design for Crowdsourcing. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, pages 165– 176, Berlin Germany, October 2018. ACM. ISBN 978-1-4503-5948-1. doi: 10.1145/3242587.3242598. URL https://dl.acm.org/doi/10.1145/3242587.3242598

    arXiv 2018

  7. [7]

    Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets

    Joseph Chee Chang, Saleema Amershi, and Ece Kamar. Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 2334–2346, Denver Colorado USA, May 2017. ACM. ISBN 978-1-4503-4655-9. doi: 10.1145/3025453.3026044. URL https://dl.acm.org/doi/10.1145/3025453.3026044

    arXiv 2017

  8. [8]

    Judgment sieve: Reducing uncertainty in group judgments through interventions targeting ambiguity versus disagreement

    Quan Ze Chen and Amy X Zhang. Judgment sieve: Reducing uncertainty in group judgments through interventions targeting ambiguity versus disagreement. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2):1–26, 2023

    work page 2023

Show all 48 references
  1. [9]

    Quan Ze Chen and Amy X. Zhang. Judgment Sieve: Reducing Uncertainty in Group Judgments through Interventions Targeting Ambiguity versus Disagreement. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2):1–26, September 2023. ISSN 2573-0142. doi: 10.1145/3610074. URL ...

    2023 arXiv

  2. [10]

    Dealing with Disagree- ments: Looking Beyond the Majority V ote in Subjective Annotations

    Aida Mostafazadeh Davani, Mark D ´ıaz, and Vinodkumar Prabhakaran. Dealing with Disagree- ments: Looking Beyond the Majority V ote in Subjective Annotations. Transactions of the As- sociation for Computational Linguistics , 10:92–110, January 2022. ISSN 2307-387X. doi: 10. 116...

    2022 doi

  3. [11]

    D3CODE: Disentan- gling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation, April 2024

    Aida Mostafazadeh Davani, Mark D´ıaz, Dylan Baker, and Vinodkumar Prabhakaran. D3CODE: Disentan- gling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation, April 2024. URL http://arxiv.org/abs/2404.10857. arXiv:2404.10857 [cs]

    2024 arXiv

  4. [12]

    Red-teaming for generative ai: Silver bullet or security theater? arXiv preprint arXiv:2401.15897, 2024

    Michael Feffer, Anusha Sinha, Zachary C Lipton, and Hoda Heidari. Red-teaming for generative ai: Silver bullet or security theater? arXiv preprint arXiv:2401.15897, 2024

    2024 arXiv

  5. [13]

    Efficient conformal prediction via cascaded inference with expanded admission

    Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Efficient conformal prediction via cascaded inference with expanded admission. arXiv preprint arXiv:2007.03114, 2020

    2007 arXiv

  6. [14]

    The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels, May 2024

    Eve Fleisig, Su Lin Blodgett, Dan Klein, and Zeerak Talat. The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels, May 2024. URL http://arxiv.org/abs/2405.05860. arXiv:2405.05860 [cs]

    2024 arXiv

  7. [15]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022

    2022 arXiv

  8. [16]

    Deep Label Distribution Learning With Label Ambiguity

    Bin-Bin Gao, Chao Xing, Chen-Wei Xie, Jianxin Wu, and Xin Geng. Deep Label Distribution Learning With Label Ambiguity. IEEE Transactions on Image Processing, 26(6):2825–2838, June 2017. ISSN 1057- 7149, 1941-0042. doi: 10.1109/TIP.2017.2689998. URL http://ieeexplore.ieee.org/d...

    2017

  9. [17]

    Label Distribution Learning

    Xin Geng. Label Distribution Learning. IEEE Transactions on Knowledge and Data Engineering , 28(7): 1734–1748, July 2016. ISSN 1041-4347. doi: 10.1109/TKDE.2016.2545658. URL http://ieeexplore. ieee.org/document/7439855/

    2016

  10. [18]

    The disagreement deconvolution: Bringing machine learning performance metrics in line with reality

    Mitchell L Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S Bernstein. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pag...

    2021

  11. [19]

    Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S

    Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S. Bernstein. The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , p...

    2021

  12. [20]

    Jury learning: Integrating dissenting voices into machine learning models

    Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing System...

    2022

  13. [21]

    Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation

    Nitesh Goyal, Ian D Kivlichan, Rachel Rosen, and Lucy Vasserman. Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):1–28, 2022

    2022

  14. [22]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 5

    2009 arXiv

  15. [23]

    Intersectionality in ai safety: Using multilevel models to understand diverse perceptions of safety in conversational ai

    Christopher Homan, Gregory Serapio-Garcia, Lora Aroyo, Mark D ´ıaz, Alicia Parrish, Vinodkumar Prabhakaran, Alex Taylor, and Ding Wang. Intersectionality in ai safety: Using multilevel models to understand diverse perceptions of safety in conversational ai. In Proceedings of t...

    2024

  16. [24]

    Culturally Aware Natural Language Inference

    Jing Huang and Diyi Yang. Culturally Aware Natural Language Inference. In Findings of the As- sociation for Computational Linguistics: EMNLP 2023 , pages 7591–7609, Singapore, 2023. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.509. URL http...

    2023 doi

  17. [25]

    Chaithanya Manam, Dwarakanath Jampani, Mariam Zaim, Meng-Han Wu, and Alexander J

    V K. Chaithanya Manam, Dwarakanath Jampani, Mariam Zaim, Meng-Han Wu, and Alexander J. Quinn. Taskmate: A mechanism to improve the quality of instructions in crowdsourcing. In Companion Proceed- ings of The 2019 World Wide Web Conference, pages 1121–1130, 2019

    2019

  18. [26]

    Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future

    Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future. Computational Linguistics, 49(1):157–198, March 2023. ISSN 0891-2017, 1530-9312. doi: 10.1162/coli a 00464. URL https://direct.mit.edu/...

    2023 doi

  19. [27]

    A Bayesian Framework for Modeling Human Evaluations

    Himabindu Lakkaraju, Jure Leskovec, Jon Kleinberg, and Sendhil Mullainathan. A Bayesian Framework for Modeling Human Evaluations. In Proceedings of the 2015 SIAM International Conference on Data Mining, pages 181–189. Society for Industrial and Applied Mathematics, June 2015. ...

    2015 doi

  20. [28]

    Reconsidering Annotator Disagreement about Racist Language: Noise or Signal?

    Savannah Larimore, Ian Kennedy, Breon Haskett, and Alina Arseniev-Koehler. Reconsidering Annotator Disagreement about Racist Language: Noise or Signal?

  21. [29]

    Learning to predict population-level label distributions

    Tong Liu, Akash Venkatachalam, Pratik Sanjay Bongale, and Christopher Homan. Learning to predict population-level label distributions. In Companion Proceedings of The 2019 World Wide Web Conference, pages 1111–1120, 2019

    2019

  22. [30]

    A safe harbor for ai evaluation and red teaming

    Shayne Longpre, Sayash Kapoor, Kevin Klyman, Ashwin Ramaswami, Rishi Bommasani, Borhane Blili- Hamelin, Yangsibo Huang, Aviya Skowron, Zheng-Xin Yong, Suhas Kotha, et al. A safe harbor for ai evaluation and red teaming. arXiv preprint arXiv:2403.04893, 2024

    2024 arXiv

  23. [31]

    A framework for automated measurement of responsible ai harms in generative ai applications

    Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, et al. A framework for automated measurement of responsible ai harms in generative ai applications. arXiv preprint arXiv:2310.17750, 2023

    arXiv

  24. [32]

    V . K. Manam and Alexander Quinn. WingIt: Efficient Refinement of Unclear Task Instructions.Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , 6:108–116, June 2018. ISSN 2769- 1349, 2769-1330. doi: 10.1609/hcomp.v6i1.13338. URL https://ojs.aaai.org/ind...

    2018 doi

  25. [33]

    Stereoset: Measuring stereotypical bias in pretrained language models

    Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456, 2020

    2004 arXiv

  26. [34]

    Diversity-aware annotation for conversational ai safety

    Alicia Parrish, Vinodkumar Prabhakaran, Lora Aroyo, Mark D´ıaz, Christopher M Homan, Greg Serapio- Garc´ıa, Alex S Taylor, and Ding Wang. Diversity-aware annotation for conversational ai safety. In Proceedings of Safety4ConvAI: The Third Workshop on Safety for Conversational A...

    2024

  27. [35]

    Inherent disagreements in human textual inferences

    Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics , 7:677–694, 2019

    2019

  28. [36]

    Inherent Disagreements in Human Textual Inferences

    Ellie Pavlick and Tom Kwiatkowski. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics , 7:677–694, November 2019. ISSN 2307-387X. doi: 10.1162/tacl a 00293. URL https://direct.mit.edu/tacl/article/43531

    2019 doi

  29. [37]

    Human uncertainty makes classification more robust

    Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9617–9626, 2019

    2019

  30. [38]

    The ’Problem’ of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation, November 2022

    Barbara Plank. The ’Problem’ of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation, November 2022. URL http://arxiv.org/abs/2211.02570. arXiv:2211.02570 [cs]

    2022 arXiv

  31. [39]

    On Releasing Annotator- Level Labels and Information in Datasets, October 2021

    Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark D ´ıaz. On Releasing Annotator- Level Labels and Information in Datasets, October 2021. URL http://arxiv.org/abs/2110.05699. arXiv:2110.05699 [cs]

    2021 arXiv

  32. [40]

    Survey Equivalence: A Procedure for Measuring Classifier Accuracy Against Human Labels, June 2021

    Paul Resnick, Yuqing Kong, Grant Schoenebeck, and Tim Weninger. Survey Equivalence: A Procedure for Measuring Classifier Accuracy Against Human Labels, June 2021. URL http://arxiv.org/abs/2106. 01254. arXiv:2106.01254 [cs]. 6

    2021 arXiv

  33. [41]

    Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. Anno- tators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection, May 2022. URL http://arxiv.org/abs/2111.07997. arXiv:2111.07997 [cs]

    2022 arXiv

  34. [42]

    Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences

    Shreya Shankar, JD Zamfirescu-Pereira, Bj¨orn Hartmann, Aditya G Parameswaran, and Ian Arawjo. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. arXiv preprint arXiv:2404.12272, 2024

    2024 arXiv

  35. [43]

    A case for soft loss functions

    Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. A case for soft loss functions. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 173–177, 2020

    2020

  36. [44]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  37. [45]

    gold data

    Ding Wang, Mark D´ıaz, Alicia Parrish, Lora Aroyo, Christopher Homan, Greg Serapio-Garc´ıa, Vinodkumar Prabhakaran, and Alex S Taylor. A case for moving beyond “gold data” in ai safety evaluation. 2024

    2024

  38. [46]

    Super- naturalinstructions:generalization via declarative instructions on 1600+ tasks

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super- naturalinstructions:generalization via declarative instructions on 1600+ tasks. In EMNLP, 2022

    2022

  39. [47]

    Disagreement Matters: Preserving Label Diversity by Jointly Modeling Item and Anno- tator Label Distributions with DisCo

    Tharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, and Christo- pher Homan. Disagreement Matters: Preserving Label Diversity by Jointly Modeling Item and Anno- tator Label Distributions with DisCo. In Findings of the Association for Computati...

    2023 doi

  40. [48]

    Many islands, many problems: An empirical examination of online safety behaviors in the caribbean

    Daricia Wilkinson and Bart Knijnenburg. Many islands, many problems: An empirical examination of online safety behaviors in the caribbean. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–25, 2022. 7

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.