{"id":"06c895e5-5737-433d-8f5a-f72377901311","arxiv_id":"1908.09091","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Fine-tuning BERT in a span-ranking coreference model raises OntoNotes F1 by 3.9 and GAP F1 by 11.5 points, with a qualitative analysis of persistent long-document and conversational errors.","lead":"This paper fine-tunes BERT for coreference resolution, the task of grouping words that refer to the same entity in a text. It reports large gains on two benchmarks and analyzes the remaining errors, informing later work on long-context and span-aware pretraining.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"One training run per configuration leaves the headline gains unquantified; seed variation could shrink the +3.9/+11.5 claims.","rationale":"Agree with the reader's weakest assumption. No internal inconsistency is apparent: Equations (1)-(6) are standard, the independent/overlap variants are clearly defined, and code plus models are released, which is independent support. The central claim is not undermined by disagreement with prior work; it fits the broader BERT trend. But the paper's entire quantitative case is single-run point estimates. The GAP result is especially exposed because of its small size and because the model is trained on OntoNotes and transferred; a difference of 11.5 F1 on a small test set could come from one low-scoring baseline run as easily as from a stable advantage. The concrete check above would settle this empirically. I therefore keep the reader's CONDITIONAL verdict; no verdict change is needed, but the authors should either add variance estimates or temper the abstract.","tokens_in":7474,"tokens_out":5476,"duration_ms":59130,"concrete_test":"Run each of the four main systems -- ELMo c2f-coref, BERT-base independent, BERT-large independent, and BERT-large overlap -- with at least 5 random seeds on OntoNotes and GAP, using the released code and official scorers. Compute the per-seed test F1 and a paired bootstrap or approximate-randomization 95% CI over documents for the BERT-large minus ELMo difference on both benchmarks. If either CI includes 0, or has lower bound below 1.0 F1, the headline effect size is not established and the abstract should be softened; if the point gap reproduces across all seeds, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that replacing the ELMo encoder in c2f-coref with fine-tuned BERT yields stable gains of +3.9 F1 (OntoNotes) and +11.5 F1 (GAP). For this to be true, the reported numbers must represent the systems rather than one favorable initialization or data-shuffling draw. Section 3 ('Implementation and Hyperparameters') specifies a single training protocol -- 20 epochs, batch size 1, random truncation to 3 or 11 segments -- but reports no seed variation and no confidence intervals in Tables 1 and 2. This matters most for GAP: the paper itself notes the dev set has only ~4,000 name-pronoun pairs, and the official metric is further split into masculine/feminine subsets, so the effective test sample is small and F1 variance is plausibly several points. The +3.9 OntoNotes gain is larger than typical seed noise for this benchmark, but absent a variance estimate we cannot separate a real effect from a lucky run, and the paper's abstract strength ('strong improvements') is calibrated to the point estimates. Table 3's qualitative claim also rests on 15 documents and hand-counted errors, but the quantitative headline is the load-bearing part.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Joshi et al. study whether replacing the ELMo/GloVe-based LSTM encoder in Lee et al.'s (2018) c2f-coref model with fine-tuned BERT improves end-to-end coreference resolution. They introduce independent and overlapping segment schemes, evaluate BERT-base and BERT-large on OntoNotes 5.0 and GAP, report absolute F1 gains of +0.9/+3.9 and +9/+11.5 over the ELMo baseline, and analyze error categories on a 15-document development subset. They also study the effect of document length and maximum segment length, and release code and models.","tokens_in":7865,"tokens_out":5943,"duration_ms":55070,"significance":"If the reported improvements are stable, the paper provides a clean, controlled demonstration that a fine-tuned transformer encoder is directly useful for coreference resolution, and the public release of code and models makes the baselines reusable. The design is well controlled (same coreference model, only encoder changed, official evaluation scripts), and the paper is appropriately cautious in attributing the remaining failures to document-level context, conversation modeling, and mention paraphrasing. The main quantitative conclusion is believable, but its current support rests entirely on single point estimates without variance information.","major_comments":[{"comment":"All headline results are reported from a single training run per configuration, with no seeds, confidence intervals, or significance tests. The paper's central claims are the absolute gains of +3.9 F1 on OntoNotes and +11.5 F1 on GAP, and the abstract uses these point estimates to claim 'strong improvements.' Since run-to-run variance, especially for the small GAP test set, could plausibly be several F1 points, the strength of the central claim is not currently supported. I ask the authors to train each configuration with at least three to five random seeds and report the mean and standard deviation, or a paired significance test, and to recalibrate the abstract's wording to the observed spread.","section":"Section 3 (Implementation and Hyperparameters), Tables 1 and 2"},{"comment":"The comparison with the concurrent EE system of Kantor and Globerson (2019) is used to state that 'the fine-tuned model seems to yield better results,' yet the difference on the official test metric is 76.9 versus 76.6, that is 0.3 F1 points. Without uncertainty estimates this comparative claim is not supported. Please either provide significance information for this specific comparison or soften the statement to a description of the observed numbers.","section":"Section 3.2 and Table 1"},{"comment":"The conclusions about the effect of document length, segment length, and the overlap variant are all drawn from the same single-run numbers. For example, the drop from max_segment_len=384 to 512 for BERT-large (77.3 to 73.6) and the lack of overlap benefit (76.9 versus 76.1 on OntoNotes test) are consistent with the authors' narrative, but without multiple runs they can be interpreted as noise. Please report variance for at least the configurations that support the main analysis claims.","section":"Section 4, Tables 4 and 5"}],"minor_comments":[{"comment":"The symbol T in 'creating a T-sized segment after every T/2 tokens' is not defined; it should be max_segment_len or an explicit segment-size parameter. Also, it is unclear from Eq. (5) whether f is a scalar gate (in which case w should be in R^{2d}) or a vector-valued gate (in which case the element-wise interpolation in Eq. (6) should be described more explicitly).","section":"Section 2.2, Eqs. (5)-(6)"},{"comment":"The qualitative analysis reports 93 versus 74 cluster-level errors on 15 documents, but the document sampling procedure and the error-category annotation protocol are not described. A short annotation protocol and inter-annotator agreement would substantially strengthen the qualitative findings.","section":"Section 4, Table 3"},{"comment":"The 'Spread' column is described as the average number of tokens between the first and last mentions in a cluster, but the text should clarify whether this is averaged over clusters within each document and then over documents; reporting standard deviations would help interpret the monotonic trend.","section":"Section 4, Table 4"},{"comment":"The sentence 'We found that this made a sizable impact of 2-3% over using the same learning rate for all parameters' reports a tuning result without showing the underlying development-set numbers; please include the comparison or a reference to a development result.","section":"Section 3 (Implementation and Hyperparameters)"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is a solid, well-written empirical paper whose main risk is the absence of variance estimates. I do not see any circularity or novelty concerns; the overlap with SpanBERT is acknowledged and the systems are compared as concurrent work. If the authors provide seed-level results, I would be comfortable accepting a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bert for Coreference is a clean, well-scoped empirical paper. It does exactly what a good baseline paper should: fixes the architecture, swaps the encoder, and reports controlled comparisons on two benchmarks. The result—BERT-large giving +3.9 F1 on OntoNotes and +11.5 on GAP over an ELMo-based c2f-coref—is believable as a real effect, though the single-run numbers deserve a raised eyebrow.\n\nWhat's new and useful: it is one of the first to fine-tune BERT inside a span-ranking coref model, and it lays out the design space: independent vs. overlapping segments, varying segment lengths, and the resulting long-context limitations. The analysis section is thoughtful—BERT-large helps with related-but-distinct entities, while conversations and mention paraphrasing remain hard. The paper also credits concurrent work (Kantor & Globerson, SpanBERT) and ships code and models. That is reproducible work.\n\nThe main soft spot is the stress-test point: no seeds, no variance estimates. On GAP, with only ~4k dev pairs, an 11.5-point swing could easily be inflated by run-to-run noise; the paper's own dev-set numbers (Tables 4 and 5) suggest the test-set gain is plausible but not conclusively separated from a lucky draw. The OntoNotes +3.9 is less worrying given the benchmark's stability. A second, minor issue: the segment-length hyperparameter is selected on dev, and the best lengths differ between base and large; that is a small HARKing risk. The qualitative analysis is only 15 documents, but it is labeled as such and supports the quantitative story rather than carrying it.\n\nOverall, the central claim holds up. I would send this to review. A serious referee should ask for multiple runs on the main configurations, or at least a calibration of expected seed noise, and a cautious rewrite of the GAP claim. But this is the kind of empirical anchor a subfield needs—clear, honest, and reproducible. I would cite it and would bring it to a reading group.","headline":"A solid, controlled baseline paper showing BERT-large helps coreference, with believable gains on OntoNotes but single-run GAP numbers that need variance estimates.","tokens_in":8226,"tokens_out":2342,"would_cite":true,"duration_ms":24480,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fine-tuning BERT inside the c2f-coref span-ranking architecture improves coreference resolution, with BERT-large gaining 3.9 F1 on OntoNotes and 11.5 F1 on GAP over an ELMo-based baseline.","keywords":["coreference resolution","BERT","fine-tuning","contextualized embeddings","span-ranking","OntoNotes","GAP","c2f-coref"],"falsifier":"Train the ELMo-based c2f-coref and the BERT-large c2f-coref models several times with different random seeds and compare the distributions of average F1 on the OntoNotes test set; the claim would be falsified if the confidence intervals for the difference straddle zero or if a single alternative seed fails to reproduce a gain near 3.9 points.","tokens_in":7322,"feed_emoji":"🔗","tokens_out":6798,"duration_ms":54444,"temperature":0.7,"pith_summary":"This paper applies BERT to coreference resolution—the task of linking expressions that refer to the same entity—and reports substantial gains over the previous ELMo-based state of the art. Replacing the LSTM encoder in the c2f-coref span-ranking model with a fine-tuned BERT transformer raises average F1 by 3.9 points on the OntoNotes benchmark and 11.5 points on the GAP benchmark when using BERT-large. A qualitative analysis shows BERT-large is particularly better than BERT-base and ELMo at distinguishing related but distinct entities (e.g., 'President' vs. 'CEO'), while still struggling with document-level context, conversations, and mention paraphrasing. The results demonstrate that pretrained language models can substantially improve coreference resolution, but the paper also shows that simply extending the segment length or using overlapping segments does not further help.","feed_headline":"BERT-large lifts coreference F1 by 3.9","feed_subtitle":"Swapping ELMo for a fine-tuned BERT in a span-ranking model also gains 11.5 points on GAP.","key_machinery":"The load-bearing machinery is the c2f-coref architecture—a span-ranking coreference model that scores mention pairs with a coarse-to-fine inference scheme and refines span representations through higher-order attention—with its LSTM encoder replaced by a fine-tuned BERT transformer. Each span is represented by the first and last BERT word-piece vectors concatenated with a learned attention vector over all word pieces in the span, and the whole model is fine-tuned end-to-end. The paper also contributes the independent and overlap segment-splitting variants for handling documents longer than BERT's 512-word input limit.","core_discovery":"On the paper's own terms, the central discovery is that fine-tuning BERT in the higher-order coreference model of Lee et al. (2018) produces strong improvements on two benchmarks: an absolute gain of 3.9 F1 over the ELMo-based c2f-coref system on the English OntoNotes 5.0 test set, and 11.5 F1 on the GAP corpus. The paper attributes the gains to BERT's richer contextualized representations, especially at the large scale. It further shows that a variant that splits documents into overlapping segments to extend context beyond BERT's 512-word limit does not improve results, and that both BERT-base and BERT-large perform worse when forced to use 512-word segments, suggesting that pretraining on longer sequences would be needed to exploit long-range context. Qualitatively, the model is better at keeping related but distinct entities separate, but still fails on conversation pronouns, entity paraphrases, and cases requiring world knowledge.","pith_inferences":["An implication left implicit is that on shorter, single-segment texts, coreference resolution has largely converged for pretrained models; the remaining headroom on documents may come not from bigger models but from architectures that model discourse structure.","The qualitative advantage of BERT-large in distinguishing related entities suggests a testable hypothesis: entity-linking or relation-aware pretraining objectives (e.g., predicting entity type or relational facts) could sharpen this ability further.","Since the paper's gains are reported from single runs, a natural follow-up is to repeat the protocol across seeds; if the F1 differences shrink or invert, the comparative claims would need revision."],"forward_implications":["If the claims hold, BERT-large plus c2f-coref sets a new state of the art on OntoNotes (76.9 Avg. F1) and GAP (85.0 F1) at the time of publication.","The large gains on GAP (paragraph-level) compared to OntoNotes (document-level) suggest that BERT's strengths are most evident when the relevant context fits in a single segment.","The failure of longer and overlapping segments implies that improving coreference on long documents will require pretraining methods that learn to use long-range context, rather than input-extension tricks.","The qualitative error analysis directly identifies three remaining research targets: document-level context, conversational dialog structure, and mention paraphrasing."],"supporting_citations":[{"why":"Supplies the BERT pretrained model and the fine-tuning recipe that the paper applies to coreference.","marker":"Devlin et al. (2019)"},{"why":"Supplies the c2f-coref higher-order span-ranking architecture that the paper extends by replacing the encoder.","marker":"Lee et al. (2018)"},{"why":"Supplies ELMo, the contextualized-embedding baseline that the BERT models are compared against.","marker":"Peters et al. (2018)"},{"why":"Provides the OntoNotes 5.0 benchmark and the CoNLL-2012 evaluation scripts used for the main results.","marker":"Pradhan et al. (2012)"},{"why":"Provides the GAP benchmark for paragraph-level gendered pronoun resolution.","marker":"Webster et al. (2018)"},{"why":"Supplies the e2e-coref predecessor span-ranking model that serves as a non-contextualized baseline.","marker":"Lee et al. (2017)"}],"fun_headline_variants":["BERT-large adds 3.9 F1 to coreference resolution","Coreference: BERT-large beats ELMo by 3.9 F1","BERT-large lifts coreference F1 3.9 points on OntoNotes","GAP up 11.5 F1 with BERT-large in coreference","BERT-large: +3.9 F1 OntoNotes, +11.5 GAP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported F1 differences come from single training runs without repeated seeds, so the central assumption is that the observed gains are stable and not artifacts of run-to-run variance.","fun_headline_variants_meta":{"raw":{"variants":["BERT-large adds 3.9 F1 to coreference resolution","Coreference: BERT-large beats ELMo by 3.9 F1","BERT-large lifts coreference F1 3.9 points on OntoNotes","GAP up 11.5 F1 with BERT-large in coreference","BERT-large: +3.9 F1 OntoNotes, +11.5 GAP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000777,"raw_usage":{"total_tokens":3373,"prompt_tokens":820,"completion_tokens":2553,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":436,"completion_tokens_details":{"reasoning_tokens":2447}},"tokens_in":436,"tokens_out":2553,"duration_ms":17586,"temperature":1.0,"reasoning_tokens":2447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:21:26.369417+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the ELMo-based c2f-coref and the BERT-large c2f-coref models several times with different random seeds and compare the distributions of average F1 on the OntoNotes test set; the claim would be falsified if the confidence intervals for the difference straddle zero or if a single alternative seed fails to reproduce a gain near 3.9 points.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the OntoNotes 5.0 benchmark and the CoNLL-2012 evaluation scripts used for the main results."}],"review_version":1}