Pith. sign in

REVIEW 3 major objections 4 minor 25 references

BERT for Coreference Resolution: Baselines and Analysis

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that fine-tuning BERT inside the c2f-coref span-ranking architecture improves coreference resolution, with BERT-large gaining 3.9 F1 on OntoNotes and 11.5 F1 on GAP over an ELMo-based baseline.

desk verdict A solid, controlled baseline paper showing BERT-large helps coreference, with believable gains on OntoNotes but single-run GAP numbers that need variance estimates. read the letter →

arxiv 1908.09091 v4 pith:XW3HMKVH submitted 2019-08-24 cs.CL

Mandar Joshi , Omer Levy , Daniel S. Weld , Luke Zettlemoyer This is my paper · ORCID
classification cs.CL
keywords coreferenceresolutionBERTfine-tuningcontextualizedembeddingsspan-rankingOntoNotesGAPc2f-coref
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper applies BERT to coreference resolution—the task of linking expressions that refer to the same entity—and reports substantial gains over the previous ELMo-based state of the art. Replacing the LSTM encoder in the c2f-coref span-ranking model with a fine-tuned BERT transformer raises average F1 by 3.9 points on the OntoNotes benchmark and 11.5 points on the GAP benchmark when using BERT-large. A qualitative analysis shows BERT-large is particularly better than BERT-base and ELMo at distinguishing related but distinct entities (e.g., 'President' vs. 'CEO'), while still struggling with document-level context, conversations, and mention paraphrasing. The results demonstrate that pretrained language models can substantially improve coreference resolution, but the paper also shows that simply extending the segment length or using overlapping segments does not further help.

What carries the argument

The load-bearing machinery is the c2f-coref architecture—a span-ranking coreference model that scores mention pairs with a coarse-to-fine inference scheme and refines span representations through higher-order attention—with its LSTM encoder replaced by a fine-tuned BERT transformer. Each span is represented by the first and last BERT word-piece vectors concatenated with a learned attention vector over all word pieces in the span, and the whole model is fine-tuned end-to-end. The paper also contributes the independent and overlap segment-splitting variants for handling documents longer than BERT's 512-word input limit.

What would settle it

Train the ELMo-based c2f-coref and the BERT-large c2f-coref models several times with different random seeds and compare the distributions of average F1 on the OntoNotes test set; the claim would be falsified if the confidence intervals for the difference straddle zero or if a single alternative seed fails to reproduce a gain near 3.9 points.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that fine-tuning BERT in the higher-order coreference model of Lee et al. (2018) produces strong improvements on two benchmarks: an absolute gain of 3.9 F1 over the ELMo-based c2f-coref system on the English OntoNotes 5.0 test set, and 11.5 F1 on the GAP corpus. The paper attributes the gains to BERT's richer contextualized representations, especially at the large scale. It further shows that a variant that splits documents into overlapping segments to extend context beyond BERT's 512-word limit does not improve results, and that both BERT-base and BERT-large perform worse when forced to use 512-word segments, suggesting that pretraining on longer sequences would be needed to exploit long-range context. Qualitatively, the model is better at keeping related but distinct entities separate, but still fails on conversation pronouns, entity paraphrases, and cases requiring world knowledge.

Load-bearing premise

The reported F1 differences come from single training runs without repeated seeds, so the central assumption is that the observed gains are stable and not artifacts of run-to-run variance.

Editorial extensions

If this is right

  • If the claims hold, BERT-large plus c2f-coref sets a new state of the art on OntoNotes (76.9 Avg. F1) and GAP (85.0 F1) at the time of publication.
  • The large gains on GAP (paragraph-level) compared to OntoNotes (document-level) suggest that BERT's strengths are most evident when the relevant context fits in a single segment.
  • The failure of longer and overlapping segments implies that improving coreference on long documents will require pretraining methods that learn to use long-range context, rather than input-extension tricks.
  • The qualitative error analysis directly identifies three remaining research targets: document-level context, conversational dialog structure, and mention paraphrasing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication left implicit is that on shorter, single-segment texts, coreference resolution has largely converged for pretrained models; the remaining headroom on documents may come not from bigger models but from architectures that model discourse structure.
  • The qualitative advantage of BERT-large in distinguishing related entities suggests a testable hypothesis: entity-linking or relation-aware pretraining objectives (e.g., predicting entity type or relational facts) could sharpen this ability further.
  • Since the paper's gains are reported from single runs, a natural follow-up is to repeat the protocol across seeds; if the F1 differences shrink or invert, the comparative claims would need revision.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. Joshi et al. study whether replacing the ELMo/GloVe-based LSTM encoder in Lee et al.'s (2018) c2f-coref model with fine-tuned BERT improves end-to-end coreference resolution. They introduce independent and overlapping segment schemes, evaluate BERT-base and BERT-large on OntoNotes 5.0 and GAP, report absolute F1 gains of +0.9/+3.9 and +9/+11.5 over the ELMo baseline, and analyze error categories on a 15-document development subset. They also study the effect of document length and maximum segment length, and release code and models.

Significance. If the reported improvements are stable, the paper provides a clean, controlled demonstration that a fine-tuned transformer encoder is directly useful for coreference resolution, and the public release of code and models makes the baselines reusable. The design is well controlled (same coreference model, only encoder changed, official evaluation scripts), and the paper is appropriately cautious in attributing the remaining failures to document-level context, conversation modeling, and mention paraphrasing. The main quantitative conclusion is believable, but its current support rests entirely on single point estimates without variance information.

major comments (3)
  1. [Section 3 (Implementation and Hyperparameters), Tables 1 and 2] All headline results are reported from a single training run per configuration, with no seeds, confidence intervals, or significance tests. The paper's central claims are the absolute gains of +3.9 F1 on OntoNotes and +11.5 F1 on GAP, and the abstract uses these point estimates to claim 'strong improvements.' Since run-to-run variance, especially for the small GAP test set, could plausibly be several F1 points, the strength of the central claim is not currently supported. I ask the authors to train each configuration with at least three to five random seeds and report the mean and standard deviation, or a paired significance test, and to recalibrate the abstract's wording to the observed spread.
  2. [Section 3.2 and Table 1] The comparison with the concurrent EE system of Kantor and Globerson (2019) is used to state that 'the fine-tuned model seems to yield better results,' yet the difference on the official test metric is 76.9 versus 76.6, that is 0.3 F1 points. Without uncertainty estimates this comparative claim is not supported. Please either provide significance information for this specific comparison or soften the statement to a description of the observed numbers.
  3. [Section 4, Tables 4 and 5] The conclusions about the effect of document length, segment length, and the overlap variant are all drawn from the same single-run numbers. For example, the drop from max_segment_len=384 to 512 for BERT-large (77.3 to 73.6) and the lack of overlap benefit (76.9 versus 76.1 on OntoNotes test) are consistent with the authors' narrative, but without multiple runs they can be interpreted as noise. Please report variance for at least the configurations that support the main analysis claims.
minor comments (4)
  1. [Section 2.2, Eqs. (5)-(6)] The symbol T in 'creating a T-sized segment after every T/2 tokens' is not defined; it should be max_segment_len or an explicit segment-size parameter. Also, it is unclear from Eq. (5) whether f is a scalar gate (in which case w should be in R^{2d}) or a vector-valued gate (in which case the element-wise interpolation in Eq. (6) should be described more explicitly).
  2. [Section 4, Table 3] The qualitative analysis reports 93 versus 74 cluster-level errors on 15 documents, but the document sampling procedure and the error-category annotation protocol are not described. A short annotation protocol and inter-annotator agreement would substantially strengthen the qualitative findings.
  3. [Section 4, Table 4] The 'Spread' column is described as the average number of tokens between the first and last mentions in a cluster, but the text should clarify whether this is averaged over clusters within each document and then over documents; reporting standard deviations would help interpret the monotonic trend.
  4. [Section 3 (Implementation and Hyperparameters)] The sentence 'We found that this made a sizable impact of 2-3% over using the same learning rate for all parameters' reports a tuning result without showing the underlying development-set numbers; please include the comparison or a reference to a development result.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all headline results are external benchmark evaluations; self-citations are concurrent and non-load-bearing.

full rationale

The paper's central claim is empirical: fine-tuning BERT inside the c2f-coref architecture improves coreference F1 on the held-out OntoNotes test set (+3.9) and on GAP (+11.5). These numbers come from running a public system against external benchmark evaluation scripts, not from any equation that reduces to the model's own fitted outputs. The method section (Section 2) modifies an existing encoder and defines the overlap interpolation in Equations 5-6, but nothing in that construction forces the reported F1 scores; the scores depend on the trained model's predictions on external data. Hyperparameters such as max segment length were selected on development sets and then applied to test sets, which is standard practice rather than circular prediction. The only self-citations are to concurrent work by the same authors (SpanBERT, RoBERTa, and the original c2f-coref); these are used as prior baselines or as an explanatory hypothesis about pretraining sequence lengths, not as the evidence that the claimed improvements happened. In particular, the Table 1 and Table 2 comparisons are against independently published baselines and official evaluation scripts, so the central result does not rely on a self-citation chain. The absence of multiple training runs and significance tests, noted in Section 3's single-protocol description, is a correctness/reproducibility concern about variance, not a circularity concern. No quoted reduction, fitted parameter renamed as prediction, or self-citation load-bearing step is present, so the appropriate score is 0.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical and depends on standard pretrained models, benchmarks, and hyperparameters tuned on the development set. No new theoretical entities are introduced.

free parameters (6)
  • max_segment_len (BERT-base) = 128
    Selected on the OntoNotes dev set; performance varies with segment length (Table 5).
  • max_segment_len (BERT-large) = 384
    Selected on the OntoNotes dev set; performance varies with segment length (Table 5).
  • learning_rate for BERT parameters = 1e-5
    Reported as making a 'sizable impact of 2-3%' over a single learning rate; chosen on dev.
  • learning_rate for task parameters = 2e-4
    Same as above.
  • dropout = 0.3
    Set in hyperparameters; no sensitivity reported.
  • training_epochs = 20
    Fixed; no early stopping described.
assumptions (3)
  • domain assumption The c2f-coref span-ranking model (Lee et al., 2018) is correctly implemented and is a valid state-of-the-art baseline.
    The paper builds directly on this model and uses it as the baseline for all comparisons (Section 2).
  • domain assumption The OntoNotes 5.0 and GAP datasets and their official evaluation scripts measure coreference quality as intended.
    All F1 numbers depend on these benchmark definitions (Section 3).
  • domain assumption BERT's pretrained weights and fine-tuning procedure are used as a black-box feature/encoder that improves over ELMo for this task.
    The paper does not analyze the internal mechanism; it assumes the comparison is fair (Section 2.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of BERT for Coreference Resolution: Baselines and Analysis." pith.science (2026). https://pith.science/paper/XW3HMKVH

@misc{pith2026190809091,
  author       = {Pith},
  title        = {Pith review of: BERT for Coreference Resolution: Baselines and Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XW3HMKVH}},
  note         = {Machine review of arXiv:1908.09091}
}
read the original abstract

We apply BERT to coreference resolution, achieving strong improvements on the OntoNotes (+3.9 F1) and GAP (+11.5 F1) benchmarks. A qualitative analysis of model predictions indicates that, compared to ELMo and BERT-base, BERT-large is particularly better at distinguishing between related but distinct entities (e.g., President and CEO). However, there is still room for improvement in modeling document-level context, conversations, and mention paraphrasing. Our code and models are publicly available.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 11 canonical work pages

  1. [1]

    Eric Bengtson and Dan Roth. 2008. http://dl.acm.org/citation.cfm?id=1613715.1613756 Understanding the value of features for coreference resolution . In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '08, pages 294--303, Stroudsburg, PA, USA. Association for Computational Linguistics

    work page arXiv 2008

  2. [2]

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. https://arxiv.org/abs/1904.10509 Generating long sequences with sparse transformers . arXiv preprint arXiv:1904.10509

    arXiv 2019

  3. [3]

    Kevin Clark and Christopher D. Manning. 2015. https://doi.org/10.3115/v1/P15-1136 Entity-centric coreference resolution with model stacking . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1405--1415. Associa...

    work page doi:10.3115/v1/p15-1136 2015

  4. [4]

    Kevin Clark and Christopher D. Manning. 2016. https://doi.org/10.18653/v1/D16-1245 Deep reinforcement learning for mention-ranking coreference models . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256--2262. Association for Computational Linguistics

    work page doi:10.18653/v1/d16-1245 2016

  5. [5]

    Pascal Denis and Jason Baldridge. 2008. http://aclweb.org/anthology/D08-1069 Specialized models and ranking for coreference resolution . In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 660--669. Association for Computational Linguistics

    work page 2008

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  7. [7]

    Greg Durrett and Dan Klein. 2013. http://aclweb.org/anthology/D13-1203 Easy victories and uphill battles in coreference resolution . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1971--1982. Association for Computational Linguistics

    work page 2013

  8. [8]

    Hongliang Fei, Xu Li, Dingcheng Li, and Ping Li. 2019. https://www.aclweb.org/anthology/P19-1064 End-to-end deep reinforcement learning based coreference resolution . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 660--665, Florence, Italy. Association for Computational Linguistics

    work page 2019

Show all 25 references
  1. [9]

    Eraldo Fernandes, C \'i cero dos Santos, and Ruy Milidi \'u . 2012. http://aclweb.org/anthology/W12-4502 Latent structure perceptron with feature induction for unrestricted coreference resolution . In Joint Conference on EMNLP and CoNLL - Shared Task, pages 41--48. Association...

    2012

  2. [10]

    Weld, Luke Zettlemoyer, and Omer Levy

    Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. https://arxiv.org/abs/1907.10529 SpanBERT : Improving pre-training by representing and predicting spans . arXiv preprint arXiv:1907.10529

    2019 arXiv

  3. [11]

    Ben Kantor and Amir Globerson. 2019. https://www.aclweb.org/anthology/P19-1066 Coreference resolution with entity equalization . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673--677, Florence, Italy. Association for Computa...

    2019

  4. [12]

    Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/D17-1018 End-to-end neural coreference resolution . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188--197, Copenhagen, Denmark. Associ...

    2017 doi

  5. [13]

    Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-2108 Higher-order coreference resolution with coarse-to-fine inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Hu...

    2018 doi

  6. [14]

    Fei Liu, Luke Zettlemoyer, and Jacob Eisenstein. 2019 a . https://arxiv.org/abs/1902.01541 The referential reader: A recurrent entity network for anaphora resolution . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5918--5925,...

    2019 arXiv

  7. [15]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 b . https://arxiv.org/abs/1907.11692 Ro BERT a: A robustly optimized BERT pretraining approach

    2019 arXiv

  8. [16]

    Sebastian Martschat and Michael Strube. 2015. https://transacl.org/ojs/index.php/tacl/article/view/604 Latent structures for coreference resolution . Transactions of the Association for Computational Linguistics, 3:405--418

    2015

  9. [17]

    Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. https://arxiv.org/abs/1708.00107 Learned in translation: Contextualized word vectors . In Advances in Neural Information Processing Systems, pages 6297--6308

    2017 arXiv

  10. [18]

    Vincent Ng and Claire Cardie. 2002. https://doi.org/10.3115/1072228.1072367 Identifying anaphoric and non-anaphoric noun phrases to improve coreference resolution . In Proceedings of the 19th International Conference on Computational Linguistics - Volume 1, COLING '02, pages 1...

    2002

  11. [19]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North American Chapter of the Associ...

    2018 doi

  12. [20]

    Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. http://aclweb.org/anthology/W12-4501 Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes . In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1--4...

    2012

  13. [21]

    Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. https://doi.org/10.1162/tacl_a_00240 Mind the GAP : A balanced corpus of gendered ambiguous pronouns . Transactions of the Association for Computational Linguistics, 6:605--617

    2018 doi

  14. [22]

    Rush, Stuart Shieber, and Jason Weston

    Sam Wiseman, Alexander M. Rush, Stuart Shieber, and Jason Weston. 2015. https://doi.org/10.3115/v1/P15-1137 Learning anaphoricity and antecedent ranking features for coreference resolution . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguis...

    2015 doi

  15. [23]

    Rush, and Stuart M

    Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. https://doi.org/10.18653/v1/N16-1114 Learning global features for coreference resolution . In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human L...

    2016 doi

  16. [24]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  17. [25]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.