WikiCREM: A Large Unsupervised Corpus for Coreference Resolution

Ana-Maria Cretu; Oana-Maria Camburu; Phil Blunsom; Thomas Lukasiewicz; Vid Kocijan; Yordan Yordanov

REVIEW 3 major objections 5 minor 104 references

WikiCREM: A Large Unsupervised Corpus for Coreference Resolution

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's central claim is that WikiCREM, a 2.4-million-example unsupervised corpus of masked personal names from Wikipedia, makes BERT-based pronoun resolution match or beat prior systems on six of seven benchmarks.

desk verdict Real dataset, solid unsupervised-learning result, but the '6 of 7 SOTA' headline overreaches: two wins come from masking the pronoun on WinoGender/WinoBias, which changes the benchmark. read the letter →

arxiv 1908.08025 v3 pith:REUU3XFM submitted 2019-08-21 cs.CL

Vid Kocijan , Oana-Maria Camburu , Ana-Maria Cretu , Yordan Yordanov , Phil Blunsom , Thomas Lukasiewicz This is my paper · ORCID

classification cs.CL

keywords coreferenceresolutionpronoununsupervisedcorpusmaskedlanguagemodelBERTfine-tuningWikipediaWinogradSchemaChallengegenderbias

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces WikiCREM, a 2.4-million-example corpus for pronoun resolution built without manual annotation by masking repeated personal names in Wikipedia passages. The authors' central claim is that fine-tuning a pre-trained language model on this corpus consistently improves pronoun resolution across seven test sets and across all three training scenarios they consider: no target data, some target data, and transductive access to unlabeled test text. On six of the seven benchmarks, the resulting models match or exceed previously reported state-of-the-art accuracy, with the best combined model reaching 78.0 F1 on GAP, 84.8% on DPR, and 86.7% on PDP. A reader should care because the corpus-generation recipe is cheap, scalable, and compatible with masked-token language models, offering a way to train hard pronoun-resolution systems without expensive hand-labelled data.

What carries the argument

The load-bearing object is WikiCREM itself, generated by a simple rule: find a one- or two-sentence Wikipedia window containing at least two distinct personal names, mask a non-first occurrence of a repeated name, and keep the example only if another candidate name precedes the mask. The predictive mechanism is masked-token scoring: a candidate name is scored by replacing the masked position with that name and reading the language model's average token log-probability, with the higher-scoring candidate chosen. During fine-tuning, BERT is trained with the negative log-likelihood of the correct name plus a max-margin term separating the correct from the incorrect candidate. This combines a cheap, fully unsupervised data-creation step with the model's existing masked-token objective, which is why the authors say the dataset slots into BERT fine-tuning with no need for manual labels.

What would settle it

Run the same WikiCREM-trained models on the original, unmasked WinoGender and WinoBias test inputs, presenting the pronoun to the model exactly as earlier systems saw it; if accuracy on those sets falls to or below the prior baselines, the reported state-of-the-art results on these two datasets are artifacts of the masking modification.

Watch

Extended reading notes

Core claim

The central claim is that masking a repeated personal name in a Wikipedia passage produces a reliable training signal for pronoun resolution: the masked occurrence stands in for a pronoun, and the task is to choose which of two candidate names fills it. WikiCREM is built by taking one- or two-sentence passages with at least two distinct personal names, masking a non-first occurrence of a repeated name, and requiring a second, distinct name to appear before the mask. Fine-tuning BERT, a bidirectional Transformer language model pre-trained on masked-token prediction, on these 2,438,897 examples with a negative-log-likelihood plus max-margin loss consistently improves accuracy on GAP, DPR, WSC, WNLI, PDP, WinoBias, and WinoGender. The best models set new state-of-the-art numbers on GAP, DPR, WinoBias, WinoGender, and PDP, while the paper counts the overall result as matching or outperforming prior systems on six of the seven datasets, the 273-problem Winograd Schema Challenge being the exception.

Load-bearing premise

The results that put WikiCREM ahead of prior systems on the gender-bias benchmarks assume that masking the pronoun so the model never sees its gender is the same task the earlier systems were scored on; if masking changes the task, those two state-of-the-art wins no longer compare like for like.

Editorial extensions

If this is right

Fine-tuning BERT on WikiCREM before any target-specific training improves accuracy on all seven test sets, so the unsupervised corpus can stand in for part of a labelled training set in low-resource settings.
Because WikiCREM also improves models that are later fine-tuned on GAP or DPR, it supplements hand-labelled data instead of only replacing it.
The generation procedure uses only name repetition in ordinary text, so it can be rerun on any sufficiently large corpus to produce more training data on demand.
The best combined model raises GAP to 78.0 F1 and DPR to 84.8% accuracy, setting new state-of-the-art numbers on those benchmarks and also on WinoBias and PDP.
Because the pronoun is masked, the model reports no gender bias on WinoGender and WinoBias by construction, while still improving accuracy on those diagnostics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The authors' own annotation finds 18 unsolvable examples in a 100-example sample, so filtering or down-weighting such examples is a natural next experiment the paper does not run.
The recipe could be applied to non-English Wikipedias or to other large text corpora; the paper mentions this possibility but does not demonstrate it.
The gains on WinoGender and WinoBias are measured with the pronoun masked out, so the model's reported zero bias reflects not seeing the gender cue rather than ignoring it; comparing against a model that sees the pronoun would separate the two.
A controlled sweep over mask position and distractor placement could identify how much of WikiCREM's value comes from the hard-example construction rule rather than from additional Wikipedia text.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: WikiCREM is a legitimate resource and the empirical core of the paper probably holds, but the headline claim does not survive contact with the evaluation details. Two of the six claimed SOTA wins are on WinoGender and WinoBias, where the paper masks the pronoun. The published SOTA numbers it compares against were produced on the original tasks, where the pronoun and its gender are visible. The paper even says these models “exhibit no bias on this dataset by design”—a statement that only makes sense if the model is not actually solving the benchmark as defined. Reporting that as SOTA is an overclaim.

What is genuinely new: the generation rule—mask repeated personal names while requiring a competing name earlier in the text—is a clear step beyond MASKED WIKI. The 63% vs 7% naturalness annotation is meaningful, and the random-masking control (BERT WIKI RAND) is the right kind of control. It shows the gains come from the masking strategy, not from extra training on Wikipedia text. The dataset and code are public, and the appendix honestly documents 18% unsolvable examples and a 95.1% annotator accuracy on the rest. Gains on GAP, DPR, PDP, WNLI, and WSC are consistent across scenarios, and even if you discount the two modified benchmarks, the corpus still helps on GAP, DPR, and WNLI.

The main soft spot is evaluation comparability. Excluding WinoGender and WinoBias, the strongest defensible claim is closer to “outperforms SOTA on GAP, DPR, and PDP; ties WNLI; below SOTA on WSC.” That is still a decent result, but it is not the advertised 6 out of 7. Second, Section 4.3 says that when fine-tuning on other datasets they retain the model with the best WikiCREM validation score, while Section 6.1 says BERT DPR uses a held-out 10% of DPR for validation. As written, these conflict and should be fixed. Smaller quibbles: GAP is transductive because BERT saw Wikipedia text at pretraining time, and hyperparameters are chosen on the WikiCREM validation set rather than the target validation sets. Both are real but not fatal; the paper mostly acknowledges the first.

The central dataset claim is supported. This paper deserves peer review. A good referee should ask the authors to redo the WinoGender/WinoBias comparisons in the original setting with the pronoun visible, or clearly relabel them as task variants, and to clarify the model-selection procedure. With those changes I would be comfortable citing it; as is, I would cite the dataset and treat the SOTA claim with caution.

Referee Report

3 major / 5 minor

Summary. The paper introduces WikiCREM, a large unsupervised corpus of 2.4 million pronoun disambiguation examples generated by masking repeated personal names in English Wikipedia. The authors fine-tune BERT on WikiCREM and evaluate across seven pronoun-resolution benchmarks in three scenarios: no target training data, additional target training data, and a transductive setting for GAP. They report consistent improvements from WikiCREM and claim to match or outperform state-of-the-art (SOTA) models on 6 of 7 datasets, including GAP, DPR, WNLI, PDP, WinoBias, and WinoGender. The corpus is released publicly, and a random-masking control (BERT WIKI RAND) is used to isolate the effect of the corpus construction from unrelated factors such as sentence length and loss function.

Significance. WikiCREM is a potentially valuable large-scale resource for pronoun resolution, and the authors provide strong evidence that fine-tuning on it improves BERT over the random-masking baseline and the raw BERT model on most benchmarks. The random-masking control is a well-designed experiment that strengthens the causal interpretation of the gains, and the manual annotation of 100 examples provides a useful quality estimate (63% natural-sounding, 18% unsolvable). However, the headline SOTA claim is substantially overstated: the evaluations on WinoGender and WinoBias use a modified task that masks the pronoun, removing the gender signal those benchmarks are designed to measure, and the GAP comparison uses a stricter scoring rule than the published SOTA. The core dataset contribution is sound, but the evaluation claims need re-scoping or re-analysis before publication.

major comments (3)

[Section 5 (WINO GENDER and WINO BIAS paragraphs) and Section 6.2] The claim of matching or outperforming SOTA on 6 of 7 datasets rests on results for WinoGender and WinoBias, but the evaluation masks the pronoun, which is a decisive change from the published benchmarks. The paper itself states in Section 5: "Our models mask the pronoun and are thus not affected by the pronoun gender. They exhibit no bias on this dataset by design." The SOTA numbers cited from Rudinger et al. (2018) and Zhao et al. (2018) are defined for the original task, where the pronoun and its gender are visible and where gender bias is measured through accuracy differences across pronoun genders and pro- vs. anti-stereotypical splits. By removing the pronoun, the model is not solving the published task, and "no bias" holds by construction. Consequently, reporting "new state-of-the-art" on WinoGender and WinoBias in Section 6.2 is not justified. If these two datasets are excluded, the claim reduces to at most: outperforms SOTA on GAP, DPR, and PDP; ties on WNLI; and is below SOTA on WSC. The authors should either re-evaluate with the pronoun visible and report bias as originally defined, or re-scope the claims to present the masked-pronoun results as a diagnostic rather than as SOTA comparisons.
[Section 5 (GAP paragraph) and Section 6.2] The GAP SOTA comparison is not like-for-like. The paper states that previous approaches count a substring match as correct, while the authors require a full-answer match, and that their Spacy NER candidate extraction fails on 18.5% of test samples, capping their achievable F1 at 91.1%. The SOTA figure of 72.1% comes from Liu et al. (2019), which may use different candidate generation and scoring. Although the stricter evaluation places the authors at a disadvantage and their higher score is therefore suggestive, the claim "outperform SOTA by 5.9%" is only meaningful if the SOTA has been re-evaluated under the same protocol. The authors should either report re-evaluated SOTA numbers under their exact protocol or clearly state the asymmetry and its direction in the text.
[Section 6.2 and Table 1] Several of the claimed improvements are within the noise of small test sets, yet the abstract and conclusion assert that fine-tuning with WikiCREM "consistently improves" performance. For example, on WNLI (145 test examples), BERT WIKI CREM improves over BERT by 1.3 points (65.8 to 67.1) and BERT WIKI CREM ALL improves over BERT ALL by 0.7 points (74.0 to 74.7). On PDP (60 examples), the improvement is 5.0 points (71.7 to 76.7), but the standard error for 60 binary examples is roughly 6 percentage points. The paper acknowledges this partially in Section 6.2 ("some of the results are subject to deviation"), but the central claim of consistency would be better supported by reporting confidence intervals, significance tests, or multiple seeds, at least for the small datasets.

minor comments (5)

[Section 3] There is a typo: "While the these sentences are not incorrect" should read "While these sentences are not incorrect".
[Throughout] The dataset name is written inconsistently as "WikiCREM" in the abstract and as "W IKI CREM" (with spurious spaces) elsewhere. Standardize the name.
[Section 4.3] The hyperparameter search is performed on a subset of 10^5 datapoints from WikiCREM, but it is not stated whether the selected hyperparameters generalize to the full dataset; consider adding a brief note or validation on the held-out WikiCREM validation set.
[Section 6.1] The definition of Scenario C (transductive) for GAP is reasonable, but the paper should clarify that the transductive setting applies only to GAP in Table 1 and not to the other datasets, to avoid ambiguity.
[Section 2] The comparison with Masked Wiki is described in terms of a 100-example manual annotation, but the annotation procedure for the naturalness estimate (63%) and its inter-annotator agreement are not reported; adding this detail would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: WikiCREM is evaluated on external benchmarks and its labels are defined by the masking construction rather than fitted to the test sets.

full rationale

The paper's central claim is empirical: fine-tuning BERT on WikiCREM improves performance across several pronoun-resolution benchmarks. WikiCREM is generated by masking repeated personal names in Wikipedia, and the correct answer is known by construction because it is the masked name; this is a data-generation label, not a parameter fitted to the evaluation sets. The evaluation is carried out on seven external benchmarks that were not used to build the corpus, and the paper includes a BERT WIKI RAND control to rule out generic effects of fine-tuning on Wikipedia text. The loss function and the WNLI rephrasing procedure are taken from the authors' own prior work (Kocijan et al., 2019), and the WNLI SOTA value in the comparison table is also from that prior paper; this is a self-citation, but it is not load-bearing for the corpus's value, because the current model's WNLI score is still computed on the external WNLI test set and the main improvement claims are anchored on GAP, DPR, and PDP, whose SOTA numbers are external. The WinoGender and WinoBias evaluations mask the pronoun, which makes 'no bias on this dataset by design' true by construction; that is a task-validity concern about comparing against published SOTA numbers, not circularity in the derivation chain, since no predicted quantity is defined in terms of an input it is supposed to explain. Overall, the paper's derivation and evaluation do not reduce to its own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four premises: BERT's pretraining is a good starting point; masked names in short Wikipedia passages are reliable training labels; Spacy NER is accurate enough for candidates; and the distribution shift does not destroy transfer. The first two are partially qualified by the paper's own measurements (18% unsolvable examples, NER failures).

free parameters (3)

alpha (max-margin coefficient) = 10
Grid-searched on a 1e5-sample subset of WikiCREM validation; controls weight of the margin loss term in Eq. (1).
beta (margin) = 0.2
Grid-searched together with alpha and learning rate; margin between correct and incorrect candidate log-probabilities.
learning_rate = 1e-5
Grid-searched over {3e-5, 1e-5, 5e-6, 3e-6}; used for all BERT fine-tuning.

assumptions (4)

domain assumption BERT-large pre-trained on Wikipedia and BooksCorpus provides representations suitable for pronoun resolution.
The entire method uses BERT as the base model and relies on its learned language statistics; the GAP evaluation is considered transductive because BERT saw Wikipedia.
domain assumption Each masked occurrence of a repeated personal name has one unambiguous correct referent among the candidates.
Dataset generation assumes the same name in a 1-2 sentence window refers to the same entity; the paper's annotation estimates 18% of examples are unsolvable, so this assumption is partially violated.
domain assumption Spacy NER identifies all personal-name candidates in source and evaluation passages.
Used both for WikiCREM construction and for candidate extraction on GAP; the paper notes NER fails for 18.5% of GAP test samples, capping achievable F1 at 91.1%.
domain assumption The distribution shift between WikiCREM and target datasets still allows useful transfer.
Section 3 reports 18% unsolvable and 63% natural-sounding examples, and Section 6.2 notes the data distribution differs from targets, yet the paper claims transfer is useful.

how reviews work

0 comments

Cite this review

Pith. "Pith review of WikiCREM: A Large Unsupervised Corpus for Coreference Resolution." pith.science (2026). https://pith.science/paper/REUU3XFM

@misc{pith2026190808025,
  author       = {Pith},
  title        = {Pith review of: WikiCREM: A Large Unsupervised Corpus for Coreference Resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/REUU3XFM}},
  note         = {Machine review of arXiv:1908.08025}
}

read the original abstract

Pronoun resolution is a major area of natural language understanding. However, large-scale training sets are still scarce, since manually labelling data is costly. In this work, we introduce WikiCREM (Wikipedia CoREferences Masked) a large-scale, yet accurate dataset of pronoun disambiguation instances. We use a language-model-based approach for pronoun resolution in combination with our WikiCREM dataset. We compare a series of models on a collection of diverse and challenging coreference resolution problems, where we match or outperform previous state-of-the-art approaches on 6 out of 7 datasets, such as GAP, DPR, WNLI, PDP, WinoBias, and WinoGender. We release our model to be used off-the-shelf for solving pronoun disambiguation.

Figures

Figures reproduced from arXiv: 1908.08025 by the authors.

**Figure 1.** WIKICREM examples. Correct answers are given in bold. We note that our dataset contains hard examples. To resolve the first example, one needs to understand that Denise was assigned a task and “meant to be the parent” thus refers to her. To resolve the second example, one needs to understand that having an abortion can only happen if one falls pregnant first. Since both candidates have feminine names, the answer c… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

104 extracted references · 77 canonical work pages

[1]

On August 2 , 2016 , it was announced that [MASK] had suffered a torn pectoral muscle and would have to undergo surgery

Throughout training camp , Jackson com- peted to be the Bengals ’ third cornerback on the depth chart against Darqueze Dennard . On August 2 , 2016 , it was announced that [MASK] had suffered a torn pectoral muscle and would have to undergo surgery . ambiguous Pronoun in place of [MASK]?: No Annotator’s answer: N/A Correct?: N/A

2016
[2]

” The [MASK] ” carried oars in the event of engine failure

The Ark ” consisted of a giant rowboat with a small engine which Beek used as his ﬁrst ferry vessel . ” The [MASK] ” carried oars in the event of engine failure . not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Ark Correct?: yes
[3]

Computing Research Repository, abs/1902.01541

The referential reader: A recurrent entity net- work for anaphora resolution. Computing Research Repository, abs/1902.01541. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc- Closky. 2014. The Stanford CoreNLP natural lan- guage processing toolkit. In Association for Compu- tational Linguistics (ACL) System...

arXiv 1902
[4]

not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Midge Ure Correct?: no

Ultravox had gone on to greater success with Midge Ure fronting the band , but when Simon left the band in 1988 , Billy Cur- rie formed a new band which later included [MASK] . not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Midge Ure Correct?: no

1988
[5]

” not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Alice Correct?: yes

The poem describes the poet ’s idyllic fam- ily life with his own three daughters , Alice , Edith , and Anne Allegra : ” grave [MASK] , and laughing Allegra , and Edith with golden hair . ” not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Alice Correct?: yes
[6]

[MASK] also described a telephone call that he received from Cho during the Thanksgiv- ing holiday break from school

Koch and Eide searched Cho ’s belongings and found a pocket knife , but they did not ﬁnd any items that they deemed threatening . [MASK] also described a telephone call that he received from Cho during the Thanksgiv- ing holiday break from school . ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A
[7]

[MASK] fought with Aragon against Castile , but helped his brother Peter , Cardinal of Foix and Arles , to crush insurgents from Aragon

However , John was able to gain the lost estates by a marriage to Joanna of Urgell , granddaughter of Peter IV of Aragon . [MASK] fought with Aragon against Castile , but helped his brother Peter , Cardinal of Foix and Arles , to crush insurgents from Aragon . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: John Correct?: yes
[8]

Until 1989 [MASK] was a district of Cagliari

Elmas , Su Masu in Sardinian language , is a ” comune ” of the Metropolitan City of Cagliari in the Italian region of Sardinia , located about northwest of Cagliari . Until 1989 [MASK] was a district of Cagliari . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Elmas Correct?: yes

1989

Show all 104 references

[9]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Zhu Correct?: yes

Later in the year , Li Keyong did send Li Sizhao and Zhou to capture Xi and Ci Prefec- tures , which had become under Zhu ’s con- trol when [MASK] conquered Huguo earlier in 901 . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Zhu Correct?: yes
[10]

The day after he returned to Hadadezer in Damascus , [MASK] suffo- cated him and seized power himself

Elisha told Hazael to tell Hadadezer that he would recover , and he revealed to Hazael that the king would recover but would die of other means . The day after he returned to Hadadezer in Damascus , [MASK] suffo- cated him and seized power himself . not ambiguous Pronoun in pl...
[11]

When Rajveer goes out of his hotel with [MASK] , he sees that they are wanted by In- terpol

As Rajveer was able to successfully lead the escape of them both , Harleen now entrusts Rajveer , subsequently falling in love with him . When Rajveer goes out of his hotel with [MASK] , he sees that they are wanted by In- terpol . not ambiguous Pronoun in place of [MASK]?: ye...
[12]

[MASK] ’s anger at Moses for not speaking to the rock on the second occasion , highlights that this is not the spiritual picture He wanted portrayed

First , God is revealed with Law , and sec- ondly , God is revealed as Person . [MASK] ’s anger at Moses for not speaking to the rock on the second occasion , highlights that this is not the spiritual picture He wanted portrayed . not ambiguous Pronoun in place of [MASK]?: yes...
[13]

” Later that day , [MASK] warned Sessum not to tell anyone what he had seen

He now said he had seen Acreman follow Cheryl Fergeson up a staircase leading to the auditorium and then heard her scream , ” No ” and ” Don ’t . ” Later that day , [MASK] warned Sessum not to tell anyone what he had seen . not ambiguous Pronoun in place of [MASK]?: no Annotat...
[14]

ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A

Meidi ﬁnally ﬁgures it out , but does not re- veal to Qi Yue and Ah Meng until [MASK] confesses . ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A
[15]

[MASK] also maintained an interest in music

In 1946 , Hill and Knowlton dissolved their partnership , and Knowlton took over the di- rection of Hill & Knowlton Cleveland , which closed shortly after Knowlton’s retirement in 1962 . [MASK] also maintained an interest in music . ambiguous Pronoun in place of [MASK]?: no An...

1946
[16]

not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Stepney Correct?: yes

Maurice White spoke to Stepney on the morning of May 17 , 1976 , but later that day , Earth , Wind & Fire keyboardist Larry Dunn received a phone call , informing him that [MASK] had died of a heart attack . not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Ste...

1976
[17]

Gao thereafter sought to submit to Tang , through[MASK]

Li Yi sought aid from Gao , who personally led two thousand cavalry soldiers to aid Li Yi , causing Dou to withdraw . Gao thereafter sought to submit to Tang , through[MASK] . not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Li Yi Correct?: yes
[18]

[MASK] started his academic career at The Queen ’s College , Oxford

At Cambridge , Rose studied under Hubert Middleton and Edward Joseph Dent from 1935 to 1939 . [MASK] started his academic career at The Queen ’s College , Oxford . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Rose Correct?: yes

1935
[19]

” Perhaps ” , [MASK] speculated , ” Leary had heard from Washington ”

Brett went back to Leary , expecting to be turned down again , but this time , Leary gave Brett the aircraft he wanted . ” Perhaps ” , [MASK] speculated , ” Leary had heard from Washington ” . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Brett Correct?: yes
[20]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Don Correct?: yes

In the reception room , a boy named Billy won ’t stop staring at Don .[MASK] is draw- ing a picture and then rips it out of his book and hands it to Billy , getting up and leaving . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Don Correct?: yes
[21]

Utkarsa was disliked and soon deposed , with a half - brother called Vijaya- malla supporting [MASK] and being at the forefront of the rebellion against the king

Hasan and Stein agree that Harsa became king in 1089 . Utkarsa was disliked and soon deposed , with a half - brother called Vijaya- malla supporting [MASK] and being at the forefront of the rebellion against the king . ambiguous Pronoun in place of [MASK]?: no Annotator’s answ...
[22]

” Therein Cherniss be- lieved Jaeger to be contrary to [MASK] , and Leisegang was unsympathetic to compatibil- ity between Plato and Aristotle in both and above

Yet this did not prevent Leisegang from re- asserting that Aristotle ’s own pattern of thinking was incompatible with a proper un- derstanding of Plato . ” Therein Cherniss be- lieved Jaeger to be contrary to [MASK] , and Leisegang was unsympathetic to compatibil- ity between ...
[23]

When Rothbaum threatens to kill them if they don ’t pay up , Bolger shoots Roth- baum ’s thugs , and Brady stabs [MASK] , killing him

Brady and Bolger leave with Rothbaum , and Rothbaum demands the money Brady owes him . When Rothbaum threatens to kill them if they don ’t pay up , Bolger shoots Roth- baum ’s thugs , and Brady stabs [MASK] , killing him . not ambiguous Pronoun in place of [MASK]?: no Annotato...
[24]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Alex Correct?: yes

As the weeks wore on , it became evident that Nick could be Lujack ’s twin brother , hence , Alex ’s son .[MASK] soon became obsessed with Nick and Mindy warned him that Alex wouldn ’t give up until she got him . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answe...
[25]

Kaneko shoots the side of [MASK] ’s leg , causing Ichi to slit Kaneko ’s throat in front of Takeshi

Due to Jijii ’s psychological manipulation , Ichi believes that Kaneko is his brother and confronts him . Kaneko shoots the side of [MASK] ’s leg , causing Ichi to slit Kaneko ’s throat in front of Takeshi . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Ic...
[26]

When Clay insults him , [MASK] shoots the Maﬁoso

Frank , Jump and other members of the gang go to Clay ’s social club , where Frank tells Clay that he wants a percentage of all Clay ’s proﬁts . When Clay insults him , [MASK] shoots the Maﬁoso . ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A
[27]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Cephiro Correct?: yes

Aska later decides not to rule Cephiro be- cause Fuu told her that the Pillar can think only of Cephiro , but since Lady Aska loves the people of Fahren , she cannot complete the task of Pillar in [MASK] . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Ceph...
[28]

[MASK] ’s reply was to ask Billings- ley for some matching socks so he would be well - dressed when he was refused admit- tance again

Billingsley ’s response was a giftbow ties for Ace . [MASK] ’s reply was to ask Billings- ley for some matching socks so he would be well - dressed when he was refused admit- tance again . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Ace Correct?: yes
[29]

[MASK] reacted by sending Ding , Ge , Pang Shigu , and Huo Cun to attack Weibo

In 890 , when Zhu asked Luo Hongxin the military governor of Weibo Circuit for per- mission to go through Luo ’s territory to at- tack Hedong , Luo refused . [MASK] reacted by sending Ding , Ge , Pang Shigu , and Huo Cun to attack Weibo . not ambiguous Pronoun in place of [MAS...
[30]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Lewis Correct?: yes

However , it is recorded that Lewis was born in 1381 and sent to the school at Oxford at age 10 ; it is also known that Chaucer ’s ” Treatise on the Astrolabe ” was written for [MASK] . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Lewis Correct?: yes
[31]

With the agreement concluded and with [MASK] giv- ing him money , horses and artillery , Sivajee set out in March 1677 for his invasions via Kurnool , Cuddapah and Madras

The spoils were to be divided between Shiv- aji , Kootab Shah and Bijapur . With the agreement concluded and with [MASK] giv- ing him money , horses and artillery , Sivajee set out in March 1677 for his invasions via Kurnool , Cuddapah and Madras . ambiguous Pronoun in place o...
[32]

In early 2016 , [MASK] signed a $1

That year , LONGi signed a contract with Yingli to cooperate on monocrystalline prod- ucts . In early 2016 , [MASK] signed a $1 . ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A

2016
[33]

Shortly before his trial begins , [MASK] kills himself

Katie is taken to Children ’s Hospital , and Louise and Wes ﬁnd themselves being ar- rested for ” what authorities are calling the worst case of child abuse they ’ve ever seen ” . Shortly before his trial begins , [MASK] kills himself . not ambiguous Pronoun in place of [MASK]...
[34]

When [MASK] comes out of the barbershop with the new hairdo , her hair is ﬂat and strange

When it becomes clear that Warren has shifted his interest from Marjorie to Bernice , Marjorie sets about humiliating Bernice by tricking her into going through with bobbing her hair . When [MASK] comes out of the barbershop with the new hairdo , her hair is ﬂat and strange . ...
[35]

At the interment , [MASK] begs Claiborne to take him and Harold on his expedition to K2 , the second highest peak in the world

Claiborne and the other survivors are rescued , thanks to quick action by Taylor and Harold . At the interment , [MASK] begs Claiborne to take him and Harold on his expedition to K2 , the second highest peak in the world . not ambiguous Pronoun in place of [MASK]?: yes Annotat...
[36]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Karen Correct?: yes

Karen overhears Bernadette talking to Keanu , who [MASK] thinks may be the baby ’s fa- ther . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Karen Correct?: yes
[37]

[MASK] has won many awards in its long history

Murray Abraham , Daphne Rubin - Vega , Henry Winkler , Kathryn Boule and Judy Kuhn also got their start with Theatreworks . [MASK] has won many awards in its long history . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Theatreworks Correct?: yes
[38]

Natasha harbors feelings of mistrust for [MASK] , who left for New York after the earthquake , and spent three years there without having ever contacted her

In Jacmel , three weeks prior to his reunion with Marie , Paul spends time with his lover and ﬁance Natasha . Natasha harbors feelings of mistrust for [MASK] , who left for New York after the earthquake , and spent three years there without having ever contacted her . not ambi...
[39]

[MASK] made his WEC debut at WEC 39 , losing to Kenji Osawa via split decision

Rebello defeated David Cho via Unanimous Decision at PXC 26 - Meanest Game Face on August 20 , 2011 . [MASK] made his WEC debut at WEC 39 , losing to Kenji Osawa via split decision . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Rebello Correct?: yes

2011
[40]

ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A

Later , Jude and Noah realize that they will be working together , as [MASK] is a new sideline reporter assigned to the Devils . ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A
[41]

Eventually , Michael , with his former mentor Tom Card helping him , tracks [MASK] ’s killer , Tyler Grey , to Panama

As such , Michael , Madeline , Sam , Fiona , and Jesse are all hell - bent on exacting revenge for Nate ’s murder . Eventually , Michael , with his former mentor Tom Card helping him , tracks [MASK] ’s killer , Tyler Grey , to Panama . not ambiguous Pronoun in place of [MASK]?...
[42]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Des Correct?: yes

Eileen later follows Des to Erinsborough to check up on him and she takes an instant dislike to [MASK] ’ housemate , Daphne Lawrence . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Des Correct?: yes
[43]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Slaughter Correct?: yes

After the match , Deuce ’n Domino attacked Snuka and Slaughter until Tony Garea and Rick Martel came into the ring to assist Snuka and [MASK] . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Slaughter Correct?: yes
[44]

Another very popular religious site of pil- grimage formerly outside the city of[MASK] but now more of a suburb is called Jamkaran

Its theological center and the Fatima Ma- sumeh Shrine are prominent features of Qom . Another very popular religious site of pil- grimage formerly outside the city of[MASK] but now more of a suburb is called Jamkaran . not ambiguous Pronoun in place of [MASK]?: yes Annotator’...
[45]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Schult Correct?: yes

Schult was married to Elke Koska for 25 years , who Schult considers his muse - she was also his manager , now in cooperation with Anna Zlotovskaya , the Russian classi- cal violinist , [MASK] married in 2010 . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer:...

2010
[46]

Shaking off [MASK] ’s pur- suit , Gordon reaches Bruce and Selina ’s hideout and collapses

Gordon wakes up and successfully escapes from Nygma . Shaking off [MASK] ’s pur- suit , Gordon reaches Bruce and Selina ’s hideout and collapses . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Nygma Correct?: yes
[47]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Leyland Correct?: yes

Although Armstrong was a third party not in privity with Leyland , and a stranger to the car purchase transaction , nonetheless Armstrong was permitted to rely on the non - derogation rights of the car owners relative to [MASK] . not ambiguous Pronoun in place of [MASK]?: yes ...
[48]

Luca lunges at Dino but [MASK] pushes him to the ground

Lana tries to intervene but is punched in the stomach by Dino . Luca lunges at Dino but [MASK] pushes him to the ground . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Dino Correct?: yes
[49]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Rapleaf Correct?: yes

On October 28 , 2010 , Facebook banned Rapleaf from scraping data on Facebook , and [MASK] said it would delete the Face- book IDs it had collected . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Rapleaf Correct?: yes

2010
[50]

[MASK] then goes into the jungle , and opens a hiding place where he is keeping Virgin Mary statues to put the one Eko gave him

After arriving at the camp , Charlie apolo- gizes to Claire , but Claire tells him to leave her and her son alone . [MASK] then goes into the jungle , and opens a hiding place where he is keeping Virgin Mary statues to put the one Eko gave him . not ambiguous Pronoun in place ...
[51]

After the war , [MASK] , Hill and a small group of former conscientious objectors created the Paciﬁca Foundation in Paciﬁca , California

During World War II , Hill , as well as Lewis , ﬁled for conscientious objector status . After the war , [MASK] , Hill and a small group of former conscientious objectors created the Paciﬁca Foundation in Paciﬁca , California . not ambiguous Pronoun in place of [MASK]?: yes An...
[52]

[MASK] and Kay ended up divorcing

Gregory , as an infant , drowned in a bathtub when Kay became distracted from a call from Sam . [MASK] and Kay ended up divorcing . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Sam Correct?: yes
[53]

It is a sequel to the 1958 ﬁlm ” [MASK] ”

Stefanie in Rio is a 1960 West German ro- mantic comedy ﬁlm directed by Curtis Bern- hardt and starring Carlos Thompson , Sabine Sinjen and Andra Parisy . It is a sequel to the 1958 ﬁlm ” [MASK] ” . ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A

1960
[54]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Kent Ling Correct?: yes

Kent Ling and his team of assassins are then forced to rescue Ling Hung , but it involved them and Ling Hung having to be in a very deadly gun battle against Kam Tin ’s hench- men and unfortunately [MASK] ’s team are all killed in the process . not ambiguous Pronoun in place o...
[55]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Sylvie Correct?: yes

When Sylvie belittles Babe , she leaves Sylvie by the canal in the rain , although Sylvie is found and Shirley realises that Babe left [MASK] to die and disowns her . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Sylvie Correct?: yes
[56]

In the season ﬁnale , [MASK] gives birth to Michael ’s son and agrees to share motherhood with the re- turning Jane

Samantha , Jennifer , Billy , Taylor and Coop leave by the end of the season . In the season ﬁnale , [MASK] gives birth to Michael ’s son and agrees to share motherhood with the re- turning Jane . ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A
[57]

[MASK] told Michael that he was not permitted on the grounds

Appears in ” ” Bucky was a worker who encountered Michael Myers as he wandered around an electrical power plant . [MASK] told Michael that he was not permitted on the grounds . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Bucky Correct?: yes
[58]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Elsie Correct?: no

In August of the same year , Laura came to Coronation Street to tell Alan and Elsie Tanner that she was getting re - married and wanted to drop [MASK] ’s loan - although Elsie refused . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Elsie Correct?: no
[59]

[MASK] was in 2nd place in the cir- cuit standings with $14 , 351 so far

On the weekend of January 14 , 2017 , Walker was planning to compete at the 2016 Montana ProRodeo Circuit Finals in Great Falls . [MASK] was in 2nd place in the cir- cuit standings with $14 , 351 so far . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Walke...

2017
[60]

Sophie Albrecht and [MASK] had similar interests and became close friends

A year later she met Friedrich Schiller and played Luise Miller in his ﬁrst performance of Kabale und Liebe . Sophie Albrecht and [MASK] had similar interests and became close friends . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Friedrich Schiller Correct?: yes
[61]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Walcott Correct?: yes

Walcott lost the count as Ali circled around a ﬂoored Liston and [MASK] tried to get him back to a neutral corner . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Walcott Correct?: yes
[62]

ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A

At the World Matchplay , Whitlock recorded wins over Kevin Painter , Raymond van Barneveld and James Wade to reach the semi - ﬁnals of the event for the second time , with [MASK] stating he was playing his best darts in ﬁve years . ambiguous Pronoun in place of [MASK]?: no Ann...
[63]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Spike Correct?: yes

When Spike lands , Jerry sticks out his tongue and throws Spike ’s lips over his own face , provoking [MASK] to chase him around the corner . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Spike Correct?: yes
[64]

In June 2002 , Armstrong and [MASK] signed a formal agreement

Armstrong felt impressed with the style of Hansen ’s work . In June 2002 , Armstrong and [MASK] signed a formal agreement . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Hansen Correct?: yes

2002
[65]

However , with Gherardo III da Camino the [MASK] regained prominence

In 1235 and 1239 the da Camino managed to obtain the rule in Treviso , but the second time they were betrayed by Alberico da Ro- mano , who expelled the Guelphs from the city . However , with Gherardo III da Camino the [MASK] regained prominence . not ambiguous Pronoun in plac...
[66]

This game was extremely important because [MASK] because Pinheiros ﬁnished the regular season in front of Flamengo precisely by direct con- frontation

The small forward Shamell Stallworth made a three - pointer with the clock reset already and gave the victory to Pinheiros . This game was extremely important because [MASK] because Pinheiros ﬁnished the regular season in front of Flamengo precisely by direct con- frontation ....
[67]

[MASK] was totally dependent on charcoal that the surrounding farmers produced

In the long period that Lars Semb was manager at Moss Jernverk he traveled al- most yearly to the mining areas and he subsequently stayed with the local agents . [MASK] was totally dependent on charcoal that the surrounding farmers produced . not ambiguous Pronoun in place of ...
[68]

The Mississippi Depart- ment of Transportation calculated an average of 14 , 000 vehicles passing along the route near [MASK]

Most trafﬁc is along the stretch between Falkner and Ripley . The Mississippi Depart- ment of Transportation calculated an average of 14 , 000 vehicles passing along the route near [MASK] . ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A
[69]

Dave and [MASK] ﬂee to a cemetery and destroy the demon with a cross

Asmodeus kills Vicki and then attacks Dave and Susan . Dave and [MASK] ﬂee to a cemetery and destroy the demon with a cross . not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Susan Correct?: yes
[70]

[MASK] and Ratsiraka met with Zafy again on June 25

In France , Andrianarivo met with former President Albert Zafy on June 11 , 2007 ; Zafy had also met with Ratsiraka and former Deputy Prime Minister Pierrot Rajaonarivelo in the previous days . [MASK] and Ratsiraka met with Zafy again on June 25 . not ambiguous Pronoun in plac...

2007
[71]

In the quarterﬁnals , for the third match in a row , [MASK] only dropped ﬁve games this time to Czech Lucie afov

In the next round , Williams faced Alona Bondarenko and once again won with only dropping 5 games . In the quarterﬁnals , for the third match in a row , [MASK] only dropped ﬁve games this time to Czech Lucie afov . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s ans...
[72]

ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A

Leornardo sent Garrett , Vasili Dassiev , Shoji Soma , and Daniel Whitehall to Giza to ac- quire a power source from a Brood vessel af- ter destroying the Brood inside .[MASK] ap- proved the idea of using the power source to run the rejuvenations chambers found by an- other te...
[73]

Shuichi and Anna start dating , much to the surprise of their friends and [MASK] ’s sister

Eventually , Saori and Yoshino rejoin Shuichi ’s group of friends , though Saori says she still hates Yoshino and Momoko . Shuichi and Anna start dating , much to the surprise of their friends and [MASK] ’s sister . ambiguous Pronoun in place of [MASK]?: no Annotator’s answer:...
[74]

not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Dzbace Correct?: yes

Gmina Branice contains the villages and set- tlements of Bliszczyce , Boboluszki , Branice , Dzbace , [MASK] - Osiedle , Dzierkowice , Grdczany , Jabonka , Jakubowice , Jdrychow- ice , Lewice , Michakowice , Niekazanice , Posucice , Turkw , Uciechowice , Wodzienin , Wdka and W...
[75]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Clinton Correct?: yes

His testimony addressed the key ” lie ” : that Clinton was allegedly pressuring Betty Cur- rie and Blumenthal himself to attest that it was Lewinsky who initially pursued[MASK] , not vice versa . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Clinton Correct?: yes
[76]

Supposedly , the group of friends met [MASK] at the park and started looking for Luis

She explained that Laura was frantic and told her that Luis jumped in the water channel and that she was unable to see him anymore . Supposedly , the group of friends met [MASK] at the park and started looking for Luis . not ambiguous Pronoun in place of [MASK]?: yes Annotator...
[77]

not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: V ere Correct?: yes

Whether it was Sidney who next challenged Vere to a duel or the other way around , Vere did not take it further , and the Queen per- sonally took Sidney to task for not recog- nizing the difference between his status and [MASK] ’s . not ambiguous Pronoun in place of [MASK]?: n...
[78]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Po Correct?: no

Wei attempts to rescue Ku , only to ﬁnd out that Po deduced Wei ’s identity as a cop , since [MASK] was too skilled compared to the rest of his gang . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Po Correct?: no
[79]

Grace joins him , wanting to talk , but Liv rushes up and demands they follow Franky and [MASK]

When she sees Franky getting into Luke ’s car , she gets into a van with Matty . Grace joins him , wanting to talk , but Liv rushes up and demands they follow Franky and [MASK] . not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Luke Correct?: yes
[80]

[MASK] stumbles outside , and discovers Danny , who is barely alive

The incapacitated Mike is stabbed repeatedly by Erin , who ties a rope around his neck , attaches the other end to a tractor , and drives the vehicle until Mike ’s neck snaps . [MASK] stumbles outside , and discovers Danny , who is barely alive . not ambiguous Pronoun in place...
[81]

[MASK] was regarded as a poor of- ﬁcer who was often drunk

Slingsby was married to the sister of Law- ford ’s wife , hence why Lawford had to give Slingsby a chance to command the Light Company to prove himself which angered Sharpe . [MASK] was regarded as a poor of- ﬁcer who was often drunk . ambiguous Pronoun in place of [MASK]?: no...
[82]

ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A

Hans and Gerda ’s mutual attraction is a chal- lenge , as Gerda is navigating her changing relationship to Lili ; but Hans ’ long - time friendship with and affection for [MASK] cause him to be supportive of both Lili and Gerda . ambiguous Pronoun in place of [MASK]?: no Annot...
[83]

There , [MASK] se- duces Sarah by challenging her to taste life

After an unsuccessful evening on the town , Clark takes Sarah to the Indian side of Cal- cutta , where they attend a party at the home of a wealthy socialite . There , [MASK] se- duces Sarah by challenging her to taste life . not ambiguous Pronoun in place of [MASK]?: yes Anno...
[84]

[MASK] has an operation and begins chemotherapy af- ter speaking to Stephanie Scully

Harold does not take the news well , but Karl eventually convinces him to ﬁght . [MASK] has an operation and begins chemotherapy af- ter speaking to Stephanie Scully . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Harold Correct?: yes
[85]

Andrew manages to save her and wound Ron ; reviving , Dale deals [MASK] a death blow

Ron knocks Dale out and leaves her in a locked car ﬁlling with exhaust , sadistically goading Andrew into braving his agorapho- bia in order to save her . Andrew manages to save her and wound Ron ; reviving , Dale deals [MASK] a death blow . not ambiguous Pronoun in place of [...
[86]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Corrigan Correct?: yes

Evans introduced two romantic interests for Corrigan : Anina Kreemar , the wealthy niece of Corrigan ’s bureau chief , and [MASK] ’s friendly rival Jennever Brand , a spirited fe- male agent of a rival clandestine spy agency . not ambiguous Pronoun in place of [MASK]?: yes Ann...
[87]

She had left [MASK] in the morning of 17 May 1940

Sirius ” sailed in ballast , having unloaded a cargo of hay at Rsneshavn after departing Troms . She had left [MASK] in the morning of 17 May 1940 . ” not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Rsneshavn Correct?: yes

1940
[88]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Rizzo Correct?: yes

Sandy tells Rizzo she plans to watch the race and offers to help [MASK] despite the ru- mors about Rizzo ’s character that have been spread around school . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Rizzo Correct?: yes
[89]

After some time , under Wenxin ’s patient persuasion , [MASK] ﬁnally agrees to send Huanhuan to a school for children with spe- cial needs

Qianru ’s mother , Fengyi tries to talk her into accepting the fact that Huanhuan is autis- tic but Qianru is unwilling to face reality . After some time , under Wenxin ’s patient persuasion , [MASK] ﬁnally agrees to send Huanhuan to a school for children with spe- cial needs ...
[90]

Trey and Alex kick [MASK] out of their home and later apologizes to Alex

After Trey pushes away Guy , Trey ﬁnally re- alized that Alex was right all along and that Guy has been trying to break them up . Trey and Alex kick [MASK] out of their home and later apologizes to Alex . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Guy C...
[91]

[MASK] was a temple priest at Hebbal tem- ple

Her ﬁfth victim was Pillama , aged 60 , and killed at Maddur Vyadyanathapura . [MASK] was a temple priest at Hebbal tem- ple . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Pillama Correct?: yes
[92]

On April 12 , Viki asks [MASK] again whether or not he loves Echo ; he says he does

When Viki asks if he loves Echo , Charlie hesitates , and Viki storms off . On April 12 , Viki asks [MASK] again whether or not he loves Echo ; he says he does . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Charlie Correct?: yes
[93]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Chen Correct?: yes

In pre - sentence proceedings , Chen ’s fa- ther , Edward Chen , was reported as saying : During his ﬁnal plea on 2 February 2006 , Chen said : On 15 February 2006 [MASK] was sentenced to life imprisonment . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Ch...

2006
[94]

Instead , [MASK] urges Andrea to escape and warn Rick and the oth- ers

Andrea considers assassinating the Governor , but Milton knows that his second - in - com- mand , Martinez , will follow through on the Governor ’s plans . Instead , [MASK] urges Andrea to escape and warn Rick and the oth- ers . not ambiguous Pronoun in place of [MASK]?: yes A...
[95]

not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Smith Correct?: yes

Burns ’s clashes with Smith was perhaps most obvious at the notorious New York City concert in 1998 where Burns attacked [MASK] after the vocalist repeatedly and de- liberately knocked one of Burns ’s cymbal stands to the ﬂoor . not ambiguous Pronoun in place of [MASK]?: yes A...

1998
[96]

not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Baldwin Correct?: yes

In any case , Baldwin ’s other brother Philip of Namur remained as regent , and eventually both of [MASK] ’s daughters , Joan and Mar- garet II , were to rule as countesses of Flan- ders . not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Baldwin Correct?: yes
[97]

In September 2014 , [MASK] sold all her interests in the Aragon

In the late 1990s , Luis Rossi , Ivan Fernan- dez , and Mercedes Fernandez purchased the Aragon . In September 2014 , [MASK] sold all her interests in the Aragon . not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Mercedes Fernandez Correct?: yes

2014
[98]

From 1953 to 1958 [MASK] was responsible for lieder , choir and church music at Radio Bremen

Peschko also played chamber music ; best known are his projects with violinist Georg Kulenkampff and cellists Enrico Mainardi and Hans Adomeit . From 1953 to 1958 [MASK] was responsible for lieder , choir and church music at Radio Bremen . not ambiguous Pronoun in place of [MA...

1953
[99]

The next meeting of the Combined Policy Committee on 15 April 1946 produced no accord on collabo- ration , and resulted in an exchange of cables between Truman and [MASK]

Patterson took the communiqu to the White House , where Truman and Attlee signed it on 16 November 1945 . The next meeting of the Combined Policy Committee on 15 April 1946 produced no accord on collabo- ration , and resulted in an exchange of cables between Truman and [MASK] ...

1945
[100]

not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Dein Correct?: yes

Dein was behind the appointment of the then little known Arsne Wenger to the manager ’s job in 1996 ; under Wenger , Arsenal have won the Premier League three times and the FA Cup seven times , and [MASK] strongly backed him and his transfer wishes through- out . not ambiguous...

1996
[103]

Hearing of this , [MASK] dismissed him from all his posts in April 1945 and ordered his arrest

Realising the war was lost , Himmler at- tempted to open peace talks with the western Allies without Hitler ’s knowledge , shortly before the end of the war . Hearing of this , [MASK] dismissed him from all his posts in April 1945 and ordered his arrest . not ambiguous Pronoun...

1945
[2012]

A fully coreference-annotated corpus of scholarly papers from the ACL Anthology.Proceed- ings of COLING 2012: Posters, pages 1059–1070. T. H. Trinh and Q. V . Le. 2018. A Simple Method for Commonsense Reasoning. Computing Research Repository, arXiv:1806.02847. Ashish Vaswani, ...

2012 arXiv
[2018]

Computing Research Repository , arXiv:1810.01375

A knowledge hunting framework for common sense reasoning. Computing Research Repository , arXiv:1810.01375. Ali Emami, Paul Trichelair, Adam Trischler, Kaheer Suleman, Hannes Schulz, and Jackie Chi Kit Che- ung. 2019. The K NOWREF Coreference Corpus: Removing gender and number...

2019 arXiv
[2019]

Computing Research Reposi- tory, arXiv:1905.06290

A surprisingly robust trick for Winograd Schema Challenge. Computing Research Reposi- tory, arXiv:1905.06290. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolu- tion based on entity-centric, p...

1905 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

On August 2 , 2016 , it was announced that [MASK] had suffered a torn pectoral muscle and would have to undergo surgery

Throughout training camp , Jackson com- peted to be the Bengals ’ third cornerback on the depth chart against Darqueze Dennard . On August 2 , 2016 , it was announced that [MASK] had suffered a torn pectoral muscle and would have to undergo surgery . ambiguous Pronoun in place of [MASK]?: No Annotator’s answer: N/A Correct?: N/A

2016

[2] [2]

” The [MASK] ” carried oars in the event of engine failure

The Ark ” consisted of a giant rowboat with a small engine which Beek used as his ﬁrst ferry vessel . ” The [MASK] ” carried oars in the event of engine failure . not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Ark Correct?: yes

[3] [3]

Computing Research Repository, abs/1902.01541

The referential reader: A recurrent entity net- work for anaphora resolution. Computing Research Repository, abs/1902.01541. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc- Closky. 2014. The Stanford CoreNLP natural lan- guage processing toolkit. In Association for Compu- tational Linguistics (ACL) System...

arXiv 1902

[4] [4]

not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Midge Ure Correct?: no

Ultravox had gone on to greater success with Midge Ure fronting the band , but when Simon left the band in 1988 , Billy Cur- rie formed a new band which later included [MASK] . not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Midge Ure Correct?: no

1988

[5] [5]

” not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Alice Correct?: yes

The poem describes the poet ’s idyllic fam- ily life with his own three daughters , Alice , Edith , and Anne Allegra : ” grave [MASK] , and laughing Allegra , and Edith with golden hair . ” not ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: Alice Correct?: yes

[6] [6]

[MASK] also described a telephone call that he received from Cho during the Thanksgiv- ing holiday break from school

Koch and Eide searched Cho ’s belongings and found a pocket knife , but they did not ﬁnd any items that they deemed threatening . [MASK] also described a telephone call that he received from Cho during the Thanksgiv- ing holiday break from school . ambiguous Pronoun in place of [MASK]?: no Annotator’s answer: N/A Correct?: N/A

[7] [7]

[MASK] fought with Aragon against Castile , but helped his brother Peter , Cardinal of Foix and Arles , to crush insurgents from Aragon

However , John was able to gain the lost estates by a marriage to Joanna of Urgell , granddaughter of Peter IV of Aragon . [MASK] fought with Aragon against Castile , but helped his brother Peter , Cardinal of Foix and Arles , to crush insurgents from Aragon . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: John Correct?: yes

[8] [8]

Until 1989 [MASK] was a district of Cagliari

Elmas , Su Masu in Sardinian language , is a ” comune ” of the Metropolitan City of Cagliari in the Italian region of Sardinia , located about northwest of Cagliari . Until 1989 [MASK] was a district of Cagliari . not ambiguous Pronoun in place of [MASK]?: yes Annotator’s answer: Elmas Correct?: yes

1989