{"id":"2c72db09-1696-431a-b085-36005ab05e4c","arxiv_id":"1908.08025","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"WikiCREM, an unsupervised 2.4M-example corpus created by masking repeated personal names in Wikipedia, improves BERT's pronoun resolution on 6 of 7 benchmarks when used for fine-tuning.","lead":"The authors built WikiCREM, a 2.4 million example training corpus for pronoun resolution by masking repeated personal names in Wikipedia text. Fine-tuning BERT on this corpus improved its performance on 6 of 7 pronoun resolution benchmarks, and the resource is released for wider use.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The '6 of 7 SOTA' claim rests on treating a masked-pronoun variant of WinoGender/WinoBias as the original benchmark, which removes the gender signal those tasks are designed to test.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the SOTA comparisons on WinoGender and WinoBias are not apples-to-apples because the paper masks the pronoun, changing the information available to the model and making the bias diagnostics vacuous by design. This concern directly undermines the ``6 out of 7'' claim, which counts WinoGender and WinoBias as two of the six wins. However, the independent value of the WikiCREM corpus is supported by the random-masking control and by consistent gains on GAP, DPR, WNLI, and other benchmarks, so the appropriate outcome is a conditional acceptance with a request to re-report the SOTA count and re-run the two gender-bias benchmarks under their original formulations. The reader's CONDITIONAL verdict is therefore appropriate; my analysis does not move the verdict, so I mark it UNCHANGED. I considered the model-selection inconsistency around ``retaining the model that performs best on the WIKI CREM validation set'' in Section 4.3, but the WinoGender/WinoBias task-equivalence problem is more load-bearing for the headline claim.","tokens_in":20334,"tokens_out":8640,"duration_ms":88883,"concrete_test":"Re-run BERT WIKI CREM DPR and BERT WIKI CREM ALL on the original, unmodified WinoGender and WinoBias test sets with the pronoun visible. Since the released model is a masked language model, score each candidate by masking the candidate mention, keeping the original pronoun token in the input, and computing log P(candidate | masked-candidate sentence with pronoun visible); select the higher-scoring candidate. Then compare the resulting accuracy and the pro/anti-stereotypical gap against the published SOTA numbers used in Table 1. If the model no longer exceeds SOTA, or if a pro/anti gap reappears, the reported \"new state-of-the-art\" and \"no bias\" claims are artifacts of the masking modification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim depends on comparability between the paper's task modification and the published WinoGender and WinoBias benchmarks. Section 5 (WinoGender) states: \"Our models mask the pronoun and are thus not affected by the pronoun gender. They exhibit no bias on this dataset by design.\" Section 6.2 then reports \"new state-of-the-art\" results on WinoGender and WinoBias. These two statements are in tension: the published SOTA numbers from Rudinger et al. (2018) and Zhao et al. (2018) are defined for the original task, where the pronoun and its gender are part of the input, and where gender bias is measured by accuracy differences across pronoun genders and across pro- vs. anti-stereotypical splits. Replacing the pronoun with [MASK] removes that signal, so \"no bias\" is true by construction and the model is not solving the published task. If WinoGender and WinoBias are excluded or re-evaluated with the pronoun visible, the headline claim \"match or outperform SOTA on 6 out of 7 datasets\" drops to at most 4/7 (GAP, DPR, and PDP outperform; WNLI ties; WSC is below SOTA). This is the load-bearing step for the SOTA portion of the central claim. The dataset's usefulness as a fine-tuning resource is separately supported by the random-masking control and by gains on GAP, DPR, and WNLI, so the concern is about the evaluation claim, not the corpus itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WikiCREM, a large unsupervised corpus of 2.4 million pronoun disambiguation examples generated by masking repeated personal names in English Wikipedia. The authors fine-tune BERT on WikiCREM and evaluate across seven pronoun-resolution benchmarks in three scenarios: no target training data, additional target training data, and a transductive setting for GAP. They report consistent improvements from WikiCREM and claim to match or outperform state-of-the-art (SOTA) models on 6 of 7 datasets, including GAP, DPR, WNLI, PDP, WinoBias, and WinoGender. The corpus is released publicly, and a random-masking control (BERT WIKI RAND) is used to isolate the effect of the corpus construction from unrelated factors such as sentence length and loss function.","tokens_in":20641,"tokens_out":8214,"duration_ms":72391,"significance":"WikiCREM is a potentially valuable large-scale resource for pronoun resolution, and the authors provide strong evidence that fine-tuning on it improves BERT over the random-masking baseline and the raw BERT model on most benchmarks. The random-masking control is a well-designed experiment that strengthens the causal interpretation of the gains, and the manual annotation of 100 examples provides a useful quality estimate (63% natural-sounding, 18% unsolvable). However, the headline SOTA claim is substantially overstated: the evaluations on WinoGender and WinoBias use a modified task that masks the pronoun, removing the gender signal those benchmarks are designed to measure, and the GAP comparison uses a stricter scoring rule than the published SOTA. The core dataset contribution is sound, but the evaluation claims need re-scoping or re-analysis before publication.","major_comments":[{"comment":"The claim of matching or outperforming SOTA on 6 of 7 datasets rests on results for WinoGender and WinoBias, but the evaluation masks the pronoun, which is a decisive change from the published benchmarks. The paper itself states in Section 5: \"Our models mask the pronoun and are thus not affected by the pronoun gender. They exhibit no bias on this dataset by design.\" The SOTA numbers cited from Rudinger et al. (2018) and Zhao et al. (2018) are defined for the original task, where the pronoun and its gender are visible and where gender bias is measured through accuracy differences across pronoun genders and pro- vs. anti-stereotypical splits. By removing the pronoun, the model is not solving the published task, and \"no bias\" holds by construction. Consequently, reporting \"new state-of-the-art\" on WinoGender and WinoBias in Section 6.2 is not justified. If these two datasets are excluded, the claim reduces to at most: outperforms SOTA on GAP, DPR, and PDP; ties on WNLI; and is below SOTA on WSC. The authors should either re-evaluate with the pronoun visible and report bias as originally defined, or re-scope the claims to present the masked-pronoun results as a diagnostic rather than as SOTA comparisons.","section":"Section 5 (WINO GENDER and WINO BIAS paragraphs) and Section 6.2"},{"comment":"The GAP SOTA comparison is not like-for-like. The paper states that previous approaches count a substring match as correct, while the authors require a full-answer match, and that their Spacy NER candidate extraction fails on 18.5% of test samples, capping their achievable F1 at 91.1%. The SOTA figure of 72.1% comes from Liu et al. (2019), which may use different candidate generation and scoring. Although the stricter evaluation places the authors at a disadvantage and their higher score is therefore suggestive, the claim \"outperform SOTA by 5.9%\" is only meaningful if the SOTA has been re-evaluated under the same protocol. The authors should either report re-evaluated SOTA numbers under their exact protocol or clearly state the asymmetry and its direction in the text.","section":"Section 5 (GAP paragraph) and Section 6.2"},{"comment":"Several of the claimed improvements are within the noise of small test sets, yet the abstract and conclusion assert that fine-tuning with WikiCREM \"consistently improves\" performance. For example, on WNLI (145 test examples), BERT WIKI CREM improves over BERT by 1.3 points (65.8 to 67.1) and BERT WIKI CREM ALL improves over BERT ALL by 0.7 points (74.0 to 74.7). On PDP (60 examples), the improvement is 5.0 points (71.7 to 76.7), but the standard error for 60 binary examples is roughly 6 percentage points. The paper acknowledges this partially in Section 6.2 (\"some of the results are subject to deviation\"), but the central claim of consistency would be better supported by reporting confidence intervals, significance tests, or multiple seeds, at least for the small datasets.","section":"Section 6.2 and Table 1"}],"minor_comments":[{"comment":"There is a typo: \"While the these sentences are not incorrect\" should read \"While these sentences are not incorrect\".","section":"Section 3"},{"comment":"The dataset name is written inconsistently as \"WikiCREM\" in the abstract and as \"W IKI CREM\" (with spurious spaces) elsewhere. Standardize the name.","section":"Throughout"},{"comment":"The hyperparameter search is performed on a subset of 10^5 datapoints from WikiCREM, but it is not stated whether the selected hyperparameters generalize to the full dataset; consider adding a brief note or validation on the held-out WikiCREM validation set.","section":"Section 4.3"},{"comment":"The definition of Scenario C (transductive) for GAP is reasonable, but the paper should clarify that the transductive setting applies only to GAP in Table 1 and not to the other datasets, to avoid ambiguity.","section":"Section 6.1"},{"comment":"The comparison with Masked Wiki is described in terms of a 100-example manual annotation, but the annotation procedure for the naturalness estimate (63%) and its inter-annotator agreement are not reported; adding this detail would improve reproducibility.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is solid and the random-masking control is a strong experimental choice, but the paper's headline claim of matching or outperforming SOTA on 6 of 7 datasets is not defensible because two of those wins rely on a modified task definition. The authors should re-scope the claims and either re-evaluate on the original tasks or present the masked-pronoun results as a separate analysis. This is fixable within the scope of the manuscript, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: WikiCREM is a legitimate resource and the empirical core of the paper probably holds, but the headline claim does not survive contact with the evaluation details. Two of the six claimed SOTA wins are on WinoGender and WinoBias, where the paper masks the pronoun. The published SOTA numbers it compares against were produced on the original tasks, where the pronoun and its gender are visible. The paper even says these models “exhibit no bias on this dataset by design”—a statement that only makes sense if the model is not actually solving the benchmark as defined. Reporting that as SOTA is an overclaim.\n\nWhat is genuinely new: the generation rule—mask repeated personal names while requiring a competing name earlier in the text—is a clear step beyond MASKED WIKI. The 63% vs 7% naturalness annotation is meaningful, and the random-masking control (BERT WIKI RAND) is the right kind of control. It shows the gains come from the masking strategy, not from extra training on Wikipedia text. The dataset and code are public, and the appendix honestly documents 18% unsolvable examples and a 95.1% annotator accuracy on the rest. Gains on GAP, DPR, PDP, WNLI, and WSC are consistent across scenarios, and even if you discount the two modified benchmarks, the corpus still helps on GAP, DPR, and WNLI.\n\nThe main soft spot is evaluation comparability. Excluding WinoGender and WinoBias, the strongest defensible claim is closer to “outperforms SOTA on GAP, DPR, and PDP; ties WNLI; below SOTA on WSC.” That is still a decent result, but it is not the advertised 6 out of 7. Second, Section 4.3 says that when fine-tuning on other datasets they retain the model with the best WikiCREM validation score, while Section 6.1 says BERT DPR uses a held-out 10% of DPR for validation. As written, these conflict and should be fixed. Smaller quibbles: GAP is transductive because BERT saw Wikipedia text at pretraining time, and hyperparameters are chosen on the WikiCREM validation set rather than the target validation sets. Both are real but not fatal; the paper mostly acknowledges the first.\n\nThe central dataset claim is supported. This paper deserves peer review. A good referee should ask the authors to redo the WinoGender/WinoBias comparisons in the original setting with the pronoun visible, or clearly relabel them as task variants, and to clarify the model-selection procedure. With those changes I would be comfortable citing it; as is, I would cite the dataset and treat the SOTA claim with caution.","headline":"Real dataset, solid unsupervised-learning result, but the '6 of 7 SOTA' headline overreaches: two wins come from masking the pronoun on WinoGender/WinoBias, which changes the benchmark.","tokens_in":21203,"tokens_out":3415,"would_cite":true,"duration_ms":32204,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that WikiCREM, a 2.4-million-example unsupervised corpus of masked personal names from Wikipedia, makes BERT-based pronoun resolution match or beat prior systems on six of seven benchmarks.","keywords":["coreference resolution","pronoun resolution","unsupervised corpus","masked language model","BERT fine-tuning","Wikipedia","Winograd Schema Challenge","gender bias"],"falsifier":"Run the same WikiCREM-trained models on the original, unmasked WinoGender and WinoBias test inputs, presenting the pronoun to the model exactly as earlier systems saw it; if accuracy on those sets falls to or below the prior baselines, the reported state-of-the-art results on these two datasets are artifacts of the masking modification.","tokens_in":20138,"feed_emoji":"🧩","tokens_out":12863,"duration_ms":113966,"temperature":0.7,"pith_summary":"The paper introduces WikiCREM, a 2.4-million-example corpus for pronoun resolution built without manual annotation by masking repeated personal names in Wikipedia passages. The authors' central claim is that fine-tuning a pre-trained language model on this corpus consistently improves pronoun resolution across seven test sets and across all three training scenarios they consider: no target data, some target data, and transductive access to unlabeled test text. On six of the seven benchmarks, the resulting models match or exceed previously reported state-of-the-art accuracy, with the best combined model reaching 78.0 F1 on GAP, 84.8% on DPR, and 86.7% on PDP. A reader should care because the corpus-generation recipe is cheap, scalable, and compatible with masked-token language models, offering a way to train hard pronoun-resolution systems without expensive hand-labelled data.","feed_headline":"Masked Wikipedia names teach BERT to resolve pronouns","feed_subtitle":"A 2.4M-example unsupervised corpus beats prior systems on six of seven coreference benchmarks.","key_machinery":"The load-bearing object is WikiCREM itself, generated by a simple rule: find a one- or two-sentence Wikipedia window containing at least two distinct personal names, mask a non-first occurrence of a repeated name, and keep the example only if another candidate name precedes the mask. The predictive mechanism is masked-token scoring: a candidate name is scored by replacing the masked position with that name and reading the language model's average token log-probability, with the higher-scoring candidate chosen. During fine-tuning, BERT is trained with the negative log-likelihood of the correct name plus a max-margin term separating the correct from the incorrect candidate. This combines a cheap, fully unsupervised data-creation step with the model's existing masked-token objective, which is why the authors say the dataset slots into BERT fine-tuning with no need for manual labels.","core_discovery":"The central claim is that masking a repeated personal name in a Wikipedia passage produces a reliable training signal for pronoun resolution: the masked occurrence stands in for a pronoun, and the task is to choose which of two candidate names fills it. WikiCREM is built by taking one- or two-sentence passages with at least two distinct personal names, masking a non-first occurrence of a repeated name, and requiring a second, distinct name to appear before the mask. Fine-tuning BERT, a bidirectional Transformer language model pre-trained on masked-token prediction, on these 2,438,897 examples with a negative-log-likelihood plus max-margin loss consistently improves accuracy on GAP, DPR, WSC, WNLI, PDP, WinoBias, and WinoGender. The best models set new state-of-the-art numbers on GAP, DPR, WinoBias, WinoGender, and PDP, while the paper counts the overall result as matching or outperforming prior systems on six of the seven datasets, the 273-problem Winograd Schema Challenge being the exception.","pith_inferences":["The authors' own annotation finds 18 unsolvable examples in a 100-example sample, so filtering or down-weighting such examples is a natural next experiment the paper does not run.","The recipe could be applied to non-English Wikipedias or to other large text corpora; the paper mentions this possibility but does not demonstrate it.","The gains on WinoGender and WinoBias are measured with the pronoun masked out, so the model's reported zero bias reflects not seeing the gender cue rather than ignoring it; comparing against a model that sees the pronoun would separate the two.","A controlled sweep over mask position and distractor placement could identify how much of WikiCREM's value comes from the hard-example construction rule rather than from additional Wikipedia text."],"forward_implications":["Fine-tuning BERT on WikiCREM before any target-specific training improves accuracy on all seven test sets, so the unsupervised corpus can stand in for part of a labelled training set in low-resource settings.","Because WikiCREM also improves models that are later fine-tuned on GAP or DPR, it supplements hand-labelled data instead of only replacing it.","The generation procedure uses only name repetition in ordinary text, so it can be rerun on any sufficiently large corpus to produce more training data on demand.","The best combined model raises GAP to 78.0 F1 and DPR to 84.8% accuracy, setting new state-of-the-art numbers on those benchmarks and also on WinoBias and PDP.","Because the pronoun is masked, the model reports no gender bias on WinoGender and WinoBias by construction, while still improving accuracy on those diagnostics."],"supporting_citations":[{"why":"Supplies BERT, the pre-trained Transformer language model whose masked-token objective the whole method builds on.","marker":"Devlin et al. (2018)"},{"why":"Provides the GAP corpus and the state-of-the-art F1 baseline used as the main transductive evaluation.","marker":"Webster et al. (2018)"},{"why":"Provides the DPR corpus used both as target training data and as a benchmark.","marker":"Rahman and Ng (2012)"},{"why":"Provides the PDP and WSC273 test sets used to evaluate Winograd-style reasoning.","marker":"Davis et al. (2017)"},{"why":"Provides the WinoGender diagnostic set and the prior baseline the paper compares against.","marker":"Rudinger et al. (2018)"},{"why":"Provides the WinoBias diagnostic set and its reported state-of-the-art baselines.","marker":"Zhao et al. (2018)"},{"why":"Supplies the prior state-of-the-art baseline on the male subset of WinoGender that the paper uses for comparison.","marker":"Durrett and Klein (2013)"},{"why":"Supplies one of the prior state-of-the-art baselines on WinoBias reported with that dataset.","marker":"Lee et al. (2017)"},{"why":"Introduces the earlier masked-Wiki dataset and the max-margin loss that WikiCREM fine-tuning inherits and extends.","marker":"Kocijan et al. (2019)"},{"why":"Defines the GLUE benchmark and the WNLI task, providing the WNLI test set evaluated in the paper.","marker":"Wang et al. (2019)"}],"fun_headline_variants":["Wikipedia name masks yield unsupervised coreference training sets","Masked names on Wikipedia: BERT learns pronouns without labels","2.4M masked Wikipedia examples beat coreference benchmarks","No annotation needed: Wikipedia masking resolves pronouns","BERT tops six of seven coreference tests via masked Wikipedia"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results that put WikiCREM ahead of prior systems on the gender-bias benchmarks assume that masking the pronoun so the model never sees its gender is the same task the earlier systems were scored on; if masking changes the task, those two state-of-the-art wins no longer compare like for like.","fun_headline_variants_meta":{"raw":{"variants":["Wikipedia name masks yield unsupervised coreference training sets","Masked names on Wikipedia: BERT learns pronouns without labels","2.4M masked Wikipedia examples beat coreference benchmarks","No annotation needed: Wikipedia masking resolves pronouns","BERT tops six of seven coreference tests via masked Wikipedia"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000303,"raw_usage":{"total_tokens":1713,"prompt_tokens":888,"completion_tokens":825,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":748}},"tokens_in":504,"tokens_out":825,"duration_ms":8566,"temperature":1.0,"reasoning_tokens":748,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:51:37.899877+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same WikiCREM-trained models on the original, unmasked WinoGender and WinoBias test inputs, presenting the pronoun to the model exactly as earlier systems saw it; if accuracy on those sets falls to or below the prior baselines, the reported state-of-the-art results on these two datasets are artifacts of the masking modification.","supporting_citations":[],"review_version":1}