{"id":"37f73884-4e25-4ad5-9517-20d0ae951670","arxiv_id":"2411.13760","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"When evaluation items admit multiple valid responses, gold-label accuracy underestimates true model performance, and the paper offers bounds on the true performance from partial knowledge.","lead":"This paper proposes a framework for evaluating large language models when tasks have more than one correct answer, called task indeterminacy. It shows that the usual gold-label evaluation undercounts model performance and suggests bounds to correct the estimate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The definitional inequality is sound, but the paper's route to estimating the valid response set—the Section 4 agreement-threshold partition bound—can conflate rater error with indeterminacy and has no stated coverage guarantee; the central practical claim rests on this unvalidated…","rationale":"The reader's weakest-assumption analysis identifies the core issue: the VRS, and therefore M*, is not directly measurable, and the paper's proposed proxy—agreement-based partitioning—is not validated. My stress-test confirms this is the most load-bearing concern. The definitional inequality M(Y_L, Y_R) <= M*(Y_L, Y_v) is sound when the gold label is an element of the VRS, and the conceptual DAG is a useful synthesis of existing work. However, the paper's practical contribution depends entirely on being able to identify or approximate which items are indeterminate. The Section 4 partition bound is presented without a formal derivation, without a statement of the conditions under which it brackets the true performance, and without an analysis of how rater error affects the threshold. The synthetic experiment in Figure 2 is described only at a high level and is not reproducible from the text, so it cannot independently support the method. The Limitations section explicitly acknowledges that the authors could not provide worked case studies or elaborate all framework components, which further supports the conditional verdict. I agree with the reader that the correct outcome is CONDITIONAL: the framework's conceptual claims are plausible but its operational estimation procedure requires explicit derivations, a specified and justified threshold-selection procedure, and validation on data where the VRS is known or can be independently established. My proposed simulation directly tests whether the agreement-threshold heuristic can recover coverage under varying rater error rates, which would settle whether the concern actually lands.","tokens_in":8251,"tokens_out":3744,"duration_ms":42349,"concrete_test":"Run a simulation with known ground-truth VRS. Fix the true fraction of indeterminate items at, say, 0.3. Simulate raters who provide the correct response with probability 1 - epsilon and otherwise choose uniformly at random from the response set, for epsilon in {0, 0.05, 0.1, 0.2}. Apply the Section 4 agreement-threshold partition bound (with a threshold chosen by the procedure described, or by cross-validation) to construct an interval for M*(Y_L, Y_v). Repeat across many trials and measure coverage: does the interval contain the true M* in at least, say, 95% of trials? If coverage degrades as epsilon increases while the true indeterminacy fraction is held fixed, the threshold heuristic does not provide a valid bound and must be replaced by an estimator that explicitly separates rater error from indeterminacy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that gold-label evaluation underestimates true performance is valid only relative to a well-defined valid response set (VRS). Section 4 defines true performance as P(Y_L in Y_v), but the VRS is not directly observed; the proposed partition bound is a heuristic that sorts items by human-rater agreement and applies a threshold below which items are deemed indeterminate. Low agreement, however, is equally consistent with exogenous rater error—careless raters, ambiguous annotation interfaces, or label noise—as with genuine multiplicity of valid responses. The paper's own DAG distinguishes Rater Unobservables and Evaluator Unobservables from the latent intent that determines the VRS, yet the partition heuristic does not disentangle these sources. No derivation in Section 4 shows that the resulting interval brackets M*(Y_L, Y_v), and no conditions on rater error rates, agreement distributions, or threshold selection are specified. If a determinate item has noisy raters, it will be misclassified as indeterminate and the interval may include responses that are not actually valid, inflating the upper bound; if an indeterminate item happens to receive high agreement because all sampled raters share one interpretation, it will be misclassified as determinate and the interval may exclude the true performance. Figure 2 is the only evidence offered for the bounds, but the synthetic data-generation process is not reported in enough detail to assess whether the figure demonstrates anything beyond the tautological case where the gold label is assumed to be an element of the VRS. Thus, even if the definitional inequality is accepted, the practical method for estimating the interval—and hence the paper's central contribution—is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that standard LLM evaluations which assume a single gold label per item are biased when the corpus contains indeterminate items, i.e., items with more than one valid response. It formalizes this with a causal DAG that distinguishes task specification, human rating processes, evaluator unobservables, and LLM responses, and it defines 'true performance' as membership in a valid response set (VRS). The authors present a synthetic experiment (Figure 2) claiming to show that gold-label evaluations underestimate true performance and that the bias grows with the proportion of indeterminate items. They also propose two methods for bounding true performance from partial knowledge: a prevalence bound and a partition bound based on human-rater agreement thresholds. The paper concludes by discussing implications for evaluation design and acknowledges limitations including lack of case studies and incomplete treatment of evaluation reliability and validity.","tokens_in":8552,"tokens_out":4368,"duration_ms":44252,"significance":"If the framework is developed rigorously, it addresses a real and timely gap: quantifying how task ambiguity and vagueness distort LLM performance estimates. The DAG-based disentanglement of task specification, rater error, and LLM behavior is a useful conceptual contribution, and the definitional point that a single gold label can understate accuracy when multiple responses are valid is correct. However, the current manuscript is a short position/workshop-style paper: the central underestimation claim is largely built into the definitions, the synthetic experiment is not reproducible from the text, the proposed bounds have no derivation or coverage guarantees, and the limitations section explicitly disclaims the empirical and methodological completeness needed for a journal-level contribution. The significance is therefore conditional on substantial further development and validation.","major_comments":[{"comment":"The claim that gold-label evaluation underestimates true performance is not established as an empirical finding; it follows directly from the definitions once one assumes that the gold label Y_R is an element of the VRS Y_v for each item. The paper never states this membership assumption or proves the inequality M(Y_L, Y_R) ≤ M*(Y_L, Y_v). As written, the 'underestimation' direction is a definitional artifact rather than a substantive result. Please state the formal conditions under which Y_R ∈ Y_v (e.g., that the rating-aggregation procedure always returns one valid response) and present the inequality as a short proposition. This would clarify what the synthetic experiment in Figure 2 actually adds beyond the definitions.","section":"§4, definition of true performance"},{"comment":"The synthetic experiment is not described in sufficient detail to assess any quantitative claim. The text says only that data were 'randomly generated consistent with our DAG'; it does not report the generative model, the number of items, the number of raters, the threshold used for the partition bound, the prevalence values, or the number of simulated trials. Figure 2 shows curves for prevalence bound, partition bound, true performance, and gold-label assumption, but there are no error bars or repeated-run variability. Consequently, the monotonic increase in bias and the relative widths of the bounds cannot be verified or reproduced. Please provide a complete generative specification (possibly in an appendix), report parameter choices and sample sizes, and include confidence intervals or repeated seeds.","section":"§4, Figure 2 and synthetic experiment"},{"comment":"The partition bound is presented only as a heuristic: sort items by human-rater (or LLM) agreement and select a threshold below which items are deemed indeterminate. No derivation shows that the resulting interval brackets M*(Y_L, Y_v), and no conditions are given on rater error rates, agreement distributions, or threshold selection. Because the paper's own DAG separates Rater Unobservables and Evaluator Unobservables from the latent intent that determines the VRS, low agreement is equally consistent with exogenous rater error as with genuine indeterminacy, and high agreement on an indeterminate item cannot be excluded. As a result, the bound has no stated coverage guarantee and may be neither an upper nor a lower bound in realistic settings. Please either provide formal conditions under which the interval contains M*(Y_L, Y_v), or reframe the proposal as an exploratory diagnostic rather than an 'error-adjusted performance interval.'","section":"§4, partition bound"},{"comment":"The Limitations section explicitly states that the framework 'does not offer a comprehensive assessment of evaluation reliability and validity,' that the authors were 'unable to provide worked case studies or framework applications,' and that the DAG 'may omit factors.' These admissions are directly relevant to the paper's central practical claim that it provides 'a method for estimating an error-adjusted performance interval.' Without case studies, a fully specified synthetic study, or a real-data application, the utility of the method for evaluation designers is not demonstrated. These limitations should be moved into the body as scoping conditions and addressed with at least one concrete application or validation before the manuscript can support its conclusions.","section":"Limitations"}],"minor_comments":[{"comment":"The text says indeterminate items are those with '|VRS| ≥ 1,' but under the definition given, determinate items have |VRS| = 1, so indeterminate items should be defined by |VRS| ≥ 2. This appears to be a typo, but it makes the partition definition incoherent and should be corrected.","section":"§4, definition of indeterminate items"},{"comment":"There are duplicate references for the same works: [8] and [9] both cite Chen and Zhang, 'Judgment Sieve'; and [18] and [19] both cite Gordon et al., 'The Disagreement Deconvolution.' Please consolidate these entries.","section":"References"},{"comment":"Figure 1 is described only as 'an overview of our causal DAG'; the right panel expansion and the red-highlighted path from Human Rating to Performance Metric to denote the gold-label approach are never explained in the text. Please add a walkthrough of the key paths and variables.","section":"Figure 1"},{"comment":"The paper uses both Y_v and Yv for the valid response set, and it is not always clear whether Y_R denotes the aggregate single gold label or the raw rating of an individual rater. Please standardize notation and define all symbols at first use.","section":"§3 and Figure 2 caption"},{"comment":"The first sentence of the Limitations section, 'One limitation of our framework (Figure 2) is that its scope is limited to forced-choice NLP tasks,' is confusing: Figure 2 is an empirical plot, not the framework. Please replace '(Figure 2)' with a reference to the relevant section or definition, or remove it.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"This is a short, workshop-style manuscript whose central idea is conceptually useful but currently underdeveloped for a journal. The direction of the bias is a definitional consequence, and the empirical and methodological support is thin: the synthetic experiment is not reproducible, the bounds have no formal guarantees, and the limitations section concedes the lack of case studies and validation. I would encourage the authors to expand the paper significantly—ideally with a formal proposition for the inequality, a fully specified synthetic study with effect sizes, and an analysis of at least one real subjective-annotation dataset—before resubmission. The duplicate references and notation inconsistencies suggest that a careful editorial pass is also needed. The paper may be a good fit for a workshops or a position-format venue, but it does not yet meet the bar for a full journal article."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"My quick take: the paper makes a clean conceptual point—if you define correctness as membership in a set of valid responses, then single-gold-label evaluation is downward-biased—but the practical estimator that would make this useful is not established. The DAG in Figure 1 is a useful synthesis of the evaluation pipeline, and the authors deserve credit for spelling out the Limitations section; they basically admit the framework is unfinished. That said, the underestimation result is a definitional consequence, not an empirical discovery: M(Ŷ_L, Ŷ_R) ≤ M*(Ŷ_L, Y_v) is true because Ŷ_R is a proxy for a singleton inside Y_v. The synthetic experiment in Figure 2 has no reported generative process or error bars, so it doesn't add much beyond illustrating the definition. The bounds are the real potential contribution, but Section 4 describes them heuristically: the partition bound sorts by agreement and thresholds, which conflates genuine indeterminacy with rater error. The paper's own DAG includes Rater Unobservables and Evaluator Unobservables, yet the partition heuristic doesn't separate them, and no coverage guarantee or derivation is given. So the interval may not bracket the true performance in practice. The Limitations section explicitly says there are no worked case studies and components are unelaborated, which matches my reading. The citation pattern looks fair and relevant; the related work on disagreement deconvolution and jury learning is appropriately positioned. Overall, this is a promising workshop paper with a sound definitional core and an honest statement of its gaps, but the method needs derivations, a reproducible experiment, and real-data validation before it can be relied on. I'd send it to peer review because the conceptual framework is worth airing and the authors have a track record of rigorous work; but I'd expect a major revision or a follow-up that fleshes out the bounds.","headline":"Clean definitional point about gold-label underestimation under task indeterminacy, but the proposed bounds are unverified heuristics and the paper needs real-data validation before the practical method is credible.","tokens_in":9091,"tokens_out":1991,"would_cite":false,"duration_ms":18478,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Scoring LLMs against a single gold label understates their true accuracy whenever tasks have more than one valid answer; the paper defines the right target as membership in the valid response set and supplies bounds to correct the bias.","keywords":["task indeterminacy","valid response set","LLM evaluation","gold label bias","human rating variation","ambiguity","vagueness","performance bounds"],"falsifier":"On a forced-choice benchmark, elicit the valid response set for every item from a diverse panel using instructions that explicitly separate 'multiple reasonable answers' from 'rater mistake,' then compute both the gold-label score and the VRS score. If the gold-label score is not strictly below the VRS score on a corpus with a substantial measured share of indeterminate items, or if the partition bound fails to contain the VRS score, the claimed bias direction and the bound procedure would both be refuted.","tokens_in":1936,"feed_emoji":"📉","tokens_out":5372,"duration_ms":113923,"temperature":0.7,"pith_summary":"LLM benchmarks usually score a model by counting how often its answer matches one pre-chosen gold label, but many tasks have no unique correct answer. This paper argues that the right target is whether the model's response lands in the valid response set, meaning the set of responses correct under at least one reasonable reading of the instruction, and defines true performance as $M^*(Y_L, Y_v) = P(Y_L \\in Y_v)$. A synthetic experiment consistent with the paper's causal model shows that the standard gold-label metric systematically underestimates this true performance, with the gap growing as the proportion of indeterminate items increases. The paper offers two ways to bound the unmeasured true score from partial knowledge: a prevalence bound based on the share of indeterminate items, and a tighter partition bound that splits the corpus into determinate and indeterminate items by human-rater agreement.","feed_headline":"Gold-label checks undercount LLM accuracy on ambiguous tasks","feed_subtitle":"When several answers are valid, scoring against one 'correct' label biases accuracy down; bounds bracket the true score.","key_machinery":"The load-bearing objects are the valid response set (VRS), defined as the set of responses correct under at least one reasonable interpretation of the instruction text, and the two performance measures $M(Y_L, Y_R) = P(Y_L = Y_R)$ versus $M^*(Y_L, Y_v) = P(Y_L \\in Y_v)$. The argument is carried by a causal directed acyclic graph of the evaluation pipeline that separates instruction text, rater identity, interpretation, rater error, and LLM response, so that indeterminacy-driven variation is distinguished from exogenous noise. On top of this, the prevalence bound and the partition bound convert partial knowledge about indeterminate items into an interval estimate of true performance, with the partition bound using an agreement-based split of the corpus into determinate and indeterminate subsets.","core_discovery":"The paper's central claim is that task indeterminacy, meaning ambiguity or vagueness that leaves more than one correct response, is not a nuisance to be averaged away but a measurable feature of the evaluation corpus, and that ignoring it biases LLM performance estimates downward. Formally, the correct performance measure is $M^*(Y_L, Y_v) = P(Y_L \\in Y_v)$, the probability that the LLM response belongs to the item's valid response set; the conventional measure $M(Y_L, Y_R) = P(Y_L = Y_R)$, based on a single aggregated gold label, counts legitimate alternative responses as errors. Since the gold label is itself one of the valid responses whenever aggregation picks a reasonable answer, the event $\\{Y_L = Y_R\\}$ is contained in $\\{Y_L \\in Y_v\\}$, so the gold-label score cannot exceed the true score, and the shortfall grows with the fraction of indeterminate items, as shown in the synthetic DAG-consistent experiment. The paper further proposes estimating this unmeasured quantity with an interval: the prevalence bound uses only the estimated proportion of indeterminate items, while the partition bound additionally labels individual items as determinate versus indeterminate, for example by an agreement threshold, and brackets true performance more tightly.","pith_inferences":["If the bias is as systematic as argued, model rankings can shift: a model that frequently produces reasonable-but-non-majority answers is under-rewarded by gold-label scoring, so leaderboard ranks are not stable under a VRS re-scoring.","A natural testable extension is to reuse existing disagreement-heavy datasets, such as safety or toxicity annotation with many raters, and check whether the partition bound, using human agreement as the split rule, brackets a VRS score measured by an independent elicitation procedure.","The framework suggests a design experiment: rewriting instructions to reduce ambiguity should shrink the gap between $M$ and $M^*$, giving evaluation designers a steerable, quantitative target for prompt and guideline refinement.","The same DAG logic applies beyond forced-choice items, which the paper explicitly leaves open; adapting the VRS to open-ended outputs would require defining a membership test for generated text, a harder but well-posed problem."],"forward_implications":["Every reported gold-label accuracy number on a corpus containing ambiguous or vague items is a lower bound on the model's true accuracy under the paper's definition, so benchmark comparisons should be read as conservative.","Evaluation designers can deploy the prevalence bound with only a small random audit of items, and the tighter partition bound by labeling items via an agreement threshold, without enumerating every valid response set.","The DAG gives a shared language for where subjectivity enters evaluation, letting designers attribute score changes to instruction text, rater identity, or model behavior rather than to a single aggregate number.","Improvements to task instructions or annotation guidelines can be quantified as reductions in the gap between the gold-label estimate and the bounded true-performance interval.","Models that choose a reasonable but non-majority response on an indeterminate item are counted as wrong by gold-label scoring and would be re-scored as correct under VRS-based evaluation."],"supporting_citations":[{"why":"Establishes the premise that AI safety evaluation should move beyond gold data, motivating the paper's definition of correctness via a set of valid responses.","marker":"[45]"},{"why":"Documents the perspectivist turn and the lack of practical frameworks for isolating sources of rating variation, which is the gap this paper fills.","marker":"[14]"},{"why":"Argues that human label variation is a core problem for ground truth in modeling and evaluation, grounding the need for a VRS-based target.","marker":"[38]"},{"why":"The disagreement-deconvolution method the paper extends; it corrects rater noise but does not model task specification, which the DAG adds.","marker":"[18]"},{"why":"Jury learning accounts for rater-identity variation but not ambiguity or vagueness; the paper's framework generalizes it.","marker":"[20]"},{"why":"Judgment Sieve is the cited crowdsourcing technique for estimating the share of indeterminate items used in the prevalence bound.","marker":"[8]"},{"why":"TaskMate is the second cited crowdsourcing technique for auditing items to estimate the proportion of indeterminacy.","marker":"[25]"},{"why":"Provides empirical evidence that rater identity shifts toxicity judgments, supporting the claim that rating variation carries meaningful signal rather than pure error.","marker":"[21]"},{"why":"Shows that human disagreements in textual inference are inherent rather than error-driven, grounding the valid-response-set concept.","marker":"[35]"}],"fun_headline_variants":["Single answer tests underrate LLMs on ambiguous tasks","Gold labels can't capture multiple right answers","When LLM tasks have many answers, scores mislead","Evaluating LLMs when no single answer is correct"],"cache_read_input_tokens":11136,"weakest_assumption_plain":"The method assumes that low human-rater agreement reliably marks an item as having multiple valid answers, rather than being a sign of sloppy or error-prone raters; if agreement cannot be separated from exogenous error, the bounds will misclassify items and may fail to bracket the true score.","fun_headline_variants_meta":{"raw":{"variants":["Single answer tests underrate LLMs on ambiguous tasks","Gold labels can't capture multiple right answers","When LLM tasks have many answers, scores mislead","Evaluating LLMs when no single answer is correct"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000335,"raw_usage":{"total_tokens":1871,"prompt_tokens":973,"completion_tokens":898,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":836}},"tokens_in":589,"tokens_out":898,"duration_ms":8591,"temperature":1.0,"reasoning_tokens":836,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:54:13.589329+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a forced-choice benchmark, elicit the valid response set for every item from a diverse panel using instructions that explicitly separate 'multiple reasonable answers' from 'rater mistake,' then compute both the gold-label score and the VRS score. If the gold-label score is not strictly below the VRS score on a corpus with a substantial measured share of indeterminate items, or if the partition bound fails to contain the VRS score, the claimed bias direction and the bound procedure would both be refuted.","supporting_citations":[{"cited_title":"gold data","cited_arxiv_id":null,"evidence_quote":"Establishes the premise that AI safety evaluation should move beyond gold data, motivating the paper's definition of correctness via a set of valid responses."},{"cited_title":"The disagreement deconvolution: Bringing machine learning performance metrics in line with reality","cited_arxiv_id":null,"evidence_quote":"The disagreement-deconvolution method the paper extends; it corrects rater noise but does not model task specification, which the DAG adds."},{"cited_title":"Jury learning: Integrating dissenting voices into machine learning models","cited_arxiv_id":null,"evidence_quote":"Jury learning accounts for rater-identity variation but not ambiguity or vagueness; the paper's framework generalizes it."},{"cited_title":"Judgment sieve: Reducing uncertainty in group judgments through interventions targeting ambiguity versus disagreement","cited_arxiv_id":null,"evidence_quote":"Judgment Sieve is the cited crowdsourcing technique for estimating the share of indeterminate items used in the prevalence bound."},{"cited_title":"Chaithanya Manam, Dwarakanath Jampani, Mariam Zaim, Meng-Han Wu, and Alexander J","cited_arxiv_id":null,"evidence_quote":"TaskMate is the second cited crowdsourcing technique for auditing items to estimate the proportion of indeterminacy."},{"cited_title":"Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation","cited_arxiv_id":null,"evidence_quote":"Provides empirical evidence that rater identity shifts toxicity judgments, supporting the claim that rating variation carries meaningful signal rather than pure error."},{"cited_title":"Inherent disagreements in human textual inferences","cited_arxiv_id":null,"evidence_quote":"Shows that human disagreements in textual inference are inherent rather than error-driven, grounding the valid-response-set concept."}],"review_version":1}