{"id":"d41414fb-72f8-4a2e-bd4b-bf4ff017308b","arxiv_id":"1908.07898","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Annotator identity functions as a shortcut in crowdsourced NLU benchmarks, and models often fail to generalize to examples from annotators absent from training.","lead":"This paper tests whether natural-language AI models learn the task behind crowdsourced benchmarks or learn the particular crowd workers who wrote the examples. The experiments show that models use annotator identity as a shortcut and generalize poorly to writers they never saw in training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Annotator-disjoint splits in §4 confound annotator identity with topic/label distribution shift; the augmentation control does not disentangle the two.","rationale":"The reader's weakest_assumption identifies exactly the concern that I consider most load-bearing: the annotator-disjoint split results are interpreted as evidence of annotator bias, but the design does not control for topic or label distribution shift. My analysis agrees with that concern and sharpens it by pointing to the augmentation experiment (Figure 3) as insufficient to resolve the confound: it distinguishes inherent difficulty from exposure, but not exposure to an annotator's style from exposure to a content topic. This is a concrete, addressable experimental design issue rather than a logical contradiction or a falsified claim. The paper still provides useful evidence that annotator identity is predictable from text and that adding annotator IDs can improve performance, though the interpretation of those results is also entangled with content shift. Because the concern can be settled with a relatively straightforward control experiment, the original CONDITIONAL verdict remains appropriate. I therefore recommend no change to the reader's verdict, while emphasizing that acceptance should require the topic-matched control (or an equivalent content-shift control) to be run and reported.","tokens_in":7204,"tokens_out":8005,"duration_ms":80031,"concrete_test":"In the OpenBookQA setup of Section 4, replace the same-annotator augmentation condition in Figure 3 with a topic-matched condition. For each held-out annotator's dev examples, draw the same number of training examples from other annotators whose question texts are nearest neighbors in BERT embedding space to the dev set's centroid, and whose label distribution matches the dev set. Keep the training-set size constant exactly as in the paper. If this topic-matched augmentation recovers the same 10–20 point gain, the Figure 3 effect is explained by topic coverage, not annotator identity. If the same-annotator augmentation yields a significantly larger gain than the topic-matched augmentation, the annotator-bias interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that models fail to generalize to new annotators because of annotator-specific style—rests on the annotator-disjoint split experiments (Table 3) and the augmentation recovery (Figure 3). The load-bearing assumption is that the performance gap between an annotator-disjoint split and a size-matched random split is caused by annotator identity, not by the mechanical content shift that the split induces. Annotators do not write random subsets of a corpus: a dev set composed of one or five annotators' examples will differ from the training set in topic distribution, vocabulary, and potentially label distribution. In OpenBookQA, where the gap reaches 23 accuracy points, each question is generated from a scientific fact; if a productive annotator concentrated on a subset of facts, the dev set is effectively a topic holdout, and the performance drop could reflect topic novelty rather than annotator style. The augmentation experiment in 'Separating annotator bias from annotator difficulty' is designed to rule out 'inherent difficulty', but it does not rule out topic exposure: moving held-out examples into training adds examples from the missing topic, which would improve performance even if no annotator-specific pattern existed. The same confound affects Section 4's Table 2: annotator IDs may correlate with label priors or topic cluster, so the accuracy gain from adding IDs does not by itself prove that the model exploits annotator style. Without a control for content distribution shift, the evidence does not establish that the generalization failures are due to annotator bias rather than to the natural content shift entailed by any annotator-based split.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether recent NLU datasets created by small numbers of crowdworkers exhibit annotator bias. Using MNLI, OpenBookQA, and CommonsenseQA with annotator IDs, the authors report three experiments: (1) adding annotator IDs as input features improves development performance (Table 2), with statistical significance for MNLI and OpenBookQA but not CommonsenseQA; (2) fine-tuned BERT can identify the most productive annotators from text, with high F1 for OpenBookQA and CommonsenseQA top annotators (Figure 2); (3) when the development set is composed of examples from annotators held out of training, performance drops relative to a size-matched random split, most dramatically in OpenBookQA multi-annotator splits (up to 23 points, Table 3). An augmentation experiment shows that moving a small fraction of held-out annotator examples into training recovers 10-20 points in OpenBookQA and CommonsenseQA, which the authors interpret as evidence that the drop is due to annotator style rather than example difficulty. The paper concludes that annotator bias should be monitored and that test-set annotators should be disjoint from training-set annotators.","tokens_in":7486,"tokens_out":5913,"duration_ms":51415,"significance":"If the results hold, the paper documents a practically important pitfall in crowdsourced NLU dataset construction: model performance on a random dev set may overestimate performance on examples produced by new annotators. The proposed remedy—disjoint annotators between train and test—is actionable and cheap to implement. The strongest evidence is the OpenBookQA multi-annotator split experiment, where all five held-out annotator sets show large negative performance differences, and the augmentation results showing rapid recovery with only a few examples. The paper also deserves credit for comparing against size-matched random splits, repeating runs to report variance, and using appropriate significance tests (McNemar/bootstrap). However, the interpretation as specifically annotator-style bias is not fully established because the split design also changes topic and label distributions.","major_comments":[{"comment":"The annotator-disjoint splits in Table 3 are not a clean manipulation of annotator identity. In OpenBookQA, each question is generated from a scientific fact, and an annotator who wrote many questions may have concentrated on a subset of facts; the held-out dev set then differs from the training set in topic composition, vocabulary, and possibly label priors, so the up-to-23-point drop could reflect a topic holdout rather than annotator-specific language style. Because the size-matched random split does not control for this content shift, Table 3 does not by itself establish that the failure is attributable to the annotator as opposed to the content distribution. I ask the authors to add a control that matches the topic/fact composition between the annotator-disjoint and random dev sets, or to otherwise measure the amount of topic overlap, and to temper the causal language accordingly.","section":"Section 4, Table 3"},{"comment":"The augmentation experiment is designed to separate 'inherent difficulty' from annotator bias, but it does not distinguish annotator style from topic exposure. When examples are moved from the held-out dev set into training, the model gains access to instances from the missing topic/label region, so performance could improve even if the original gap had nothing to do with the annotator's idiolect. The control of keeping the training set size constant does not address this. A stronger control would be to add the same number of examples from the same topics written by other annotators, or to compare augmentation by held-out-annotator examples against augmentation by topic-matched examples from seen annotators.","section":"Section 4, Figure 3 (Separating annotator bias from annotator difficulty)"},{"comment":"The text states that adding the annotator ID 'improves model performance across all datasets,' but the CommonsenseQA difference is not statistically significant (p = 0.1198). This should be reported as not significant, and the claim in the abstract that models improve with annotator identifiers should be qualified. The other two datasets are significant, so the overall point survives, but the current wording overstates the evidence.","section":"Section 4, Table 2"},{"comment":"The CommonsenseQA multi-annotator results are largely within noise: only one of the five sets shows a clear negative difference (-9.5 ± 8.3), while two are positive. The conclusion that annotator bias is evident in 'two of these datasets' (Section 5) is therefore too strong for CommonsenseQA unless an aggregate test across the five sets is provided. Reporting a combined p-value or effect size would clarify whether CommonsenseQA supports the claim.","section":"Section 4, Table 3 and Section 5"}],"minor_comments":[{"comment":"There are formatting inconsistencies in dataset names: 'OPEN BOOK QA.' has a stray period, and 'O PEN BOOK QA' and 'C OMMONSENSE QA' contain extra spaces. Please fix these throughout.","section":"Section 3"},{"comment":"The caption notes that OpenBookQA has only four data points because the second annotator is not in the original development set; this limitation should be stated in the main text, since it affects the interpretation of the recognition performance.","section":"Figure 2 caption"},{"comment":"The p-values are reported in a needlessly confusing notation (e.g., '1.83e−2' and '11.98e−2'); use conventional decimals such as 0.018 and 0.120.","section":"Table 2"},{"comment":"The description does not specify whether evaluation after moving k% of the dev set into training is performed on the remaining dev examples or on the full original dev set; please clarify, and state the number of random seeds used for the repeats.","section":"Section 4, augmentation experiment"}],"recommendation":"major_revision","confidential_remarks":"The paper's central result is likely to be of interest, but the interpretation should be reframed as distribution shift associated with annotators rather than proven stylistic bias unless the topic/label confound is addressed. The recommended remedy of disjoint train/test annotators is sensible regardless of the mechanism. If new experiments are not feasible, the authors should at minimum soften the causal claims and acknowledge the confound explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know up front. First, this paper introduces a genuinely new measurement toolkit for annotator bias: using annotator IDs as input features, measuring how well models recognize productive annotators, and evaluating disjoint-annotator splits. Second, the paper's lead interpretation—that models fail to generalize to unseen annotators because of annotator-specific style—is partly confounded with content distribution shift, and the authors' control does not fully disentangle them.\n\nThe novelty is real. Prior work on annotation artifacts (Gururangan et al. 2018, Poliak et al. 2018) showed models exploit shallow cues, but nobody had made annotator identity itself a feature or tested held-out annotator splits. The experiments are clearly designed, standard BERT fine-tuning, and they report McNemar/bootstrap tests. The OpenBookQA results are the strongest: held-out annotator splits drop by up to 23 accuracy points relative to random splits, and augmenting training with a small portion of the target annotator's examples recovers 10–20 points. The recognition experiment (Figure 2) is another nice contribution—models identify the most productive annotators from text with high F1, direct evidence of annotator-specific signal.\n\nThe weak spot is causal interpretation of the held-out splits. As the authors acknowledge in Section 4, annotators do not write random subsets of the corpus. In OpenBookQA each question is generated from a scientific fact, so a dev set consisting of one annotator's questions is likely a topic holdout as much as an annotator holdout. The augmentation experiment is designed to separate 'inherent difficulty' from bias, but moving dev examples into training also adds examples of the missing topic, so it does not rule out topic exposure. The CommonsenseQA ID-gain is not significant (p = 0.1198), and MNLI's results are mixed—so the general claim rests mainly on OpenBookQA. Also missing: code and data release, and the top-5 annotator threshold is arbitrary.\n\nWho should read this: anyone building or evaluating crowdsourced NLU benchmarks. The practical recommendation—keep test-set annotators disjoint from training annotators—is sensible regardless of the confound. I would send this to review, not desk-reject, but ask for a control that matches label or topic distributions between annotator-disjoint and random splits, or for softened causal claims.","headline":"Solid empirical study with a useful new measurement apparatus for annotator bias; just don't take the held-out-split interpretation as proof of style over content shift.","tokens_in":8031,"tokens_out":3213,"would_cite":true,"duration_ms":29346,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Crowdsourced NLU datasets carry a hidden signal: who wrote each example, and models use that signal.","keywords":["annotator bias","crowdsourcing","natural language understanding","dataset construction","generalization","annotator recognition","BERT","annotation artifacts"],"falsifier":"A concrete test is to rebuild one of the datasets so that each underlying item is rewritten by several annotators, then compare accuracy on an annotator-disjoint split against a random split over versions while holding topic and label distribution fixed. If the gap disappears under this matching, the bias is not about annotator identity; if the gap persists even though the same items are written by different people, the paper's conclusion is confirmed.","tokens_in":7008,"feed_emoji":"📝","tokens_out":8149,"duration_ms":69462,"temperature":0.7,"pith_summary":"This paper sets out to show that crowdsourced natural language understanding (NLU) datasets contain a systematic bias tied to the annotator who wrote each example. Across three recent datasets, the authors find that models perform better when given annotator IDs as input features, that models can often recognize the most productive annotators from the text alone, and that in two of the three datasets models generalize poorly to examples written by annotators not seen in training. The central consequence is that reported accuracies may partly reflect how well a model imitates familiar annotators rather than how well it understands language. If correct, dataset creators should track annotator identity during collection and use annotator-disjoint test splits to obtain a truer measure of generalization.","feed_headline":"Unseen annotators cost up to 23 accuracy points in QA","feed_subtitle":"Models learn who wrote each example; held-out writers drop accuracy by up to 23 points.","key_machinery":"The load-bearing mechanism is the annotator-disjoint train/test split, in which no annotator appears in both the training and development sets; the gap between such splits and random splits of identical size is taken as a direct measure of annotator bias. Supporting machinery includes adding the annotator ID as an input token to a pre-trained transformer, fine-tuning the model to predict annotator identity as a separate classification task, and an augmentation procedure that moves small fractions of a held-out annotator's examples into training to separate style effects from intrinsic difficulty.","core_discovery":"The paper claims that annotator bias—systematic linguistic patterns tied to the worker who produced an example—is present in current NLU datasets and inflates model performance. Evidence comes from three experiments: prepending a textual annotator ID to each training example raises development accuracy on all three datasets (significant for OpenBookQA and MNLI); a model fine-tuned to predict the top-five annotators from an example's text reaches F1 scores as high as 0.91, showing that annotator style is readable; and when training and development sets are split so that annotators are disjoint, development accuracy drops by up to 23 points in OpenBookQA and is often lower in CommonsenseQA, while MNLI shows smaller and mixed effects. An augmentation experiment—moving small fractions of a held-out annotator's examples into training—quickly recovers most of the lost performance in the QA datasets, suggesting the drop reflects annotator style rather than example difficulty.","pith_inferences":["The same audit method—annotator-ID benefit, recognition probes, and disjoint splits—could be applied to newer large-scale NLU datasets to check whether they carry annotator style.","If annotator bias is widespread, comparing models across papers may be confounded by which annotators happen to land in training versus test; public benchmarks might need to report annotator-holdout scores as a standard metric.","A further study could test whether adding annotator IDs encodes topic or label preferences rather than pure lexical style, which would change the choice of mitigation: decorrelating topics from annotators versus aggregating multiple annotators per item.","The augmentation result suggests a possible data-collection strategy: periodically recruit a few fresh annotators, measure the performance gap on their examples, and use that gap as an early warning during dataset construction."],"forward_implications":["Reported accuracies on crowdsourced NLU benchmarks can overstate true understanding, because models may exploit annotator-specific wording patterns.","Dataset creators should test on new annotators during data collection to detect annotator bias early.","Holding out entire annotators from the training set would give a more honest estimate of generalization to unseen users.","Adding a small number of examples from a held-out annotator to training can quickly restore performance, suggesting a practical remedy when bias is found.","Annotator recognition accuracy is a simple diagnostic: if a model can tell who wrote an example, the dataset is likely carrying style signals."],"supporting_citations":[{"why":"Provides the MNLI dataset, one of the three testbeds for the annotator-bias experiments.","marker":"(Williams et al., 2018)"},{"why":"Provides the OpenBookQA dataset, where annotator-disjoint splits show the largest accuracy drops.","marker":"(Mihaylov et al., 2018)"},{"why":"Provides the CommonsenseQA dataset, where annotator splits underperform random splits in several configurations.","marker":"(Talmor et al., 2019)"},{"why":"Supplies the pre-trained BERT model used for all reported experiments.","marker":"(Devlin et al., 2018)"},{"why":"Guides the selection of significance tests used to compare model performance with and without annotator IDs.","marker":"(Dror et al., 2018)"},{"why":"Provides the bootstrap significance test used for the QA datasets.","marker":"(Berg-Kirkpatrick et al., 2012)"},{"why":"Provides the paired significance test used for MNLI.","marker":"(McNemar, 1947)"}],"fun_headline_variants":["Annotator split costs up to 23 points in QA","Models learn annotator style, not just the task","Annotator ID predicts examples: F1 up to 0.91","Held-out annotators drop NLU accuracy","Annotator bias: A hidden flaw in NLU benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The interpretation assumes that the performance gap between annotator-disjoint and random splits is caused by annotator identity (writing style) rather than by incidental differences in topic, difficulty, or label distribution between the split groups.","fun_headline_variants_meta":{"raw":{"variants":["Annotator split costs up to 23 points in QA","Models learn annotator style, not just the task","Annotator ID predicts examples: F1 up to 0.91","Held-out annotators drop NLU accuracy","Annotator bias: A hidden flaw in NLU benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001126,"raw_usage":{"total_tokens":4651,"prompt_tokens":883,"completion_tokens":3768,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":3684}},"tokens_in":499,"tokens_out":3768,"duration_ms":26640,"temperature":1.0,"reasoning_tokens":3684,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:53:47.951523+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test is to rebuild one of the datasets so that each underlying item is rewritten by several annotators, then compare accuracy on an annotator-disjoint split against a random split over versions while holding topic and label distribution fixed. If the gap disappears under this matching, the bias is not about annotator identity; if the gap persists even though the same items are written by different people, the paper's conclusion is confirmed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the OpenBookQA dataset, where annotator-disjoint splits show the largest accuracy drops."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Guides the selection of significance tests used to compare model performance with and without annotator IDs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the bootstrap significance test used for the QA datasets."}],"review_version":1}