Pith. sign in

REVIEW 5 major objections 5 minor 24 references

Improving Generalization in Coreference Resolution via Adversarial Training

T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adversarial training on span representations improves coreference resolution on original, renamed, and GAP test sets.

desk verdict A useful no-leakage evaluation protocol, but the claimed benefit of adversarial training is not established because the comparison baseline is a single off-the-shelf checkpoint. read the letter →

arxiv 1908.04728 v1 pith:4PN3GEMG submitted 2019-08-13 cs.CL

Sanjay Subramanian , Dan Roth This is my paper · ORCID
classification cs.CL
keywords coreferenceresolutionadversarialtrainingfastgradientsignmethodgeneralizationno-leakageevaluationnamedentityreplacementGAPdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the leading neural coreference resolver generalizes beyond the names it saw in training, and reports that it does not: replacing every person and geopolitical name in the CoNLL test set by a name absent from training lowers CoNLL F1 by more than one point. The authors then retrain the same architecture with adversarial gradient training applied to its span representations, and report higher F1 on the unchanged CoNLL test set (73.23 vs 72.96), on the no-leakage test set (72.32 vs 71.84), and on the GAP pronoun dataset (74.7 vs 73.3). The point of the paper is that a generic robustness technique can push an already-strong coreference system to generalize to unseen names, and that perturbing intermediate span representations, rather than input embeddings, is the effective place to do it.

What carries the argument

The load-bearing mechanism is adversarial training at the span-representation layer. A span representation is the model's vector for a candidate mention: the concatenation of the contextualized vectors at the span's start and end, an attention-weighted sum of its word vectors, and a learned width embedding, so that the model's predictions and loss depend on the input text only through these vectors. The paper computes the gradient of the coreference loss with respect to each span representation, creates a perturbed span vector by adding a small normalized step $\epsilon g_i/\|g_i\|$ in the gradient direction, and trains on $\mathcal{L}_{\mathrm{total}} = \alpha \mathcal{L} + (1-\alpha)\mathcal{L}_{\mathrm{adv}}$ with $\alpha = 0.6$ and $\epsilon = 1$. The second piece of machinery is the evaluation protocol itself: a no-leakage test set built by replacing PER names with gender-matched names from census and Wikipedia gazetteer lists and GPE names with GeoNames names of the same location category, removing every name seen in training.

What would settle it

Re-run the no-leakage evaluation many times with fresh random replacement names drawn from the same or independent name lists and check whether the baseline's drop and the adversarially trained model's gain persist across draws. If ranking flips or the gap shrinks to noise under re-sampling, the reported advantage is an artifact of one particular replacement sample rather than a property of adversarial training.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes two claims. First, the current best CoNLL-trained system is vulnerable to name leakage: when PER and GPE mentions are replaced by gender-matched or location-category-matched names that never occur in training, performance on the CoNLL test set drops by more than 1 F1. Second, retraining that exact system with the fast-gradient-sign method applied at the span-representation layer fixes most of this vulnerability and improves the original test score as well, yielding 73.23 vs 72.96 on the original CoNLL set, 72.32 vs 71.84 on the no-leakage set, and 74.7 vs 73.3 on GAP. The paper therefore claims a new state of the art on all three test sets. The key is not a new architecture but a changed training objective on the same architecture.

Load-bearing premise

The load-bearing premise is that swapping a person's name for another gender-matched name and a place for a same-category place never changes the gold coreference structure, so any measured drop is caused by unfamiliar names rather than by artifacts of the replacement.

Editorial extensions

If this is right

  • Retraining with the span-level adversarial objective raises F1 on the unchanged CoNLL test set, so the method acts as a regularizer rather than only a defense against renamed inputs.
  • The larger relative gain on the no-leakage set indicates the trained system relies less on memorized PER and GPE names and more on structural cues like gender and location category.
  • The GAP improvement shows the benefit transfers to Wikipedia text and to a pronoun-to-name binary decision task, not just to the CoNLL genre mix.
  • The adversarial step leaves the GAP male/female F1 ratio essentially unchanged, so the gain does not come at the cost of increased gender bias, and it does not reduce that bias either.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit is an ablation over $\alpha$ and $\epsilon$; a narrow band of useful perturbation sizes would suggest margin hardening, while a flat response would suggest the gain is mostly extra noise.
  • The no-leakage protocol could become a standard 'unseen entity' evaluation for span-based models, since it changes one axis of variation while keeping genre and annotation fixed.
  • If the perturbation is truly acting on span representations, the same training objective should improve robustness on other span-labeling problems, such as named entity recognition or relation extraction, and that is a direct empirical check of the mechanism.
  • Because the reported numbers come from a single random draw of replacement names, the paper's point estimates should be read as one sample; averaging over multiple draws would give a more stable estimate of the method's advantage.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper studies how coreference resolution generalizes to unseen names. The authors construct a "No Leakage" version of the CoNLL-2012 test set by replacing PER and GPE named entities with names not seen in training, controlling for gender and location category. They apply FGSM adversarial training to the span representations of the Lee et al. (2018) end-to-end coreference model, minimizing alpha*L + (1-alpha)*L_adv with alpha=0.6 and epsilon=1. They report that the Lee et al. system drops by more than 1 F1 under the No Leakage modification (71.84 vs 72.96) and that the adversarially trained model improves over the original model on the original CoNLL test set (73.23 vs 72.96), on the No Leakage set (72.32 vs 71.84), and on the GAP dataset (74.7 vs 73.3 overall F1), concluding that the adversarially trained system is a new state of the art.

Significance. If the results hold, this paper would demonstrate a simple and computationally cheap regularization that improves coreference generalization and transfers across datasets, and the No Leakage diagnostic is a useful contribution. The paper ships a code link and uses public benchmarks; the GAP improvements are statistically significant (McNemar p<0.05) and directionally consistent across gender. However, the central causal claim is not fully supported because the comparison baseline is a released checkpoint rather than a retrained control, the CoNLL margins are small and not significant at conventional levels, and the No Leakage assumption that replacements preserve gold structure is unvalidated. These issues are addressable with additional experiments.

major comments (5)
  1. [§5, Tables 3 and 4] The central comparison is confounded by the baseline protocol. The adversarially trained model is trained in-house for 355,000 iterations, while the Lee et al. (2018) baseline is the released pretrained checkpoint (Section 5). A single released checkpoint is one draw from a stochastic training process, and the reported margins (0.27 and 0.48 CoNLL F1; 1.4 GAP F1) are of the same magnitude as run-to-run variation from initialization, data order, and hardware nondeterminism. To support the claim that the adversarial loss causes the improvement, the authors must retrain the non-adversarial Lee et al. model under identical code, hyperparameters, data ordering, and multiple seeds, and report the mean and spread. This affects every reported result, including the GAP claim.
  2. [§5.1, Table 3] The CoNLL improvements are not established statistically. The caption uses p<0.21 as the significance threshold for underlining, which is far weaker than the conventional p<0.05, and no exact p-values or confidence intervals are reported. With differences of 0.27 and 0.48 F1, the "consistently performs better" claim for CoNLL is unsupported. Please report exact p-values, confidence intervals (or bootstrap), and ideally multiple runs.
  3. [§4, §4.1, §4.2] The No Leakage evaluation assumes that replacing PER and GPE names does not change true coreference structure, but this assumption is not validated. Gender and location-type controls do not account for all ways in which a name change can alter a document's coreference (e.g., nominal predicates, titles, generic or metonymic uses, number/gender cues beyond pronouns, and parser head errors). If replacements introduce artifacts, the measured degradation and the apparent benefit of adversarial training could be evaluation artifacts. Please validate the replacement protocol, for example by human annotation of a sample of modified documents, and report an error analysis.
  4. [§3] Hyperparameter selection is underreported. alpha=0.6 and epsilon=1 are said to "work well" without a search criterion or sensitivity analysis, and the claim that perturbing character embeddings "was not as effective" is not accompanied by numbers. Because the central claim depends on the adversarial objective, please provide a small sensitivity analysis and, crucially, an ablation that trains the same architecture without the adversarial term under the identical protocol (this also addresses the baseline issue).
  5. [§5.2, Table 4] The GAP comparison may not be fully controlled. The system outputs must be converted to binary (pronoun, name) predictions, and the authors note that their Lee et al. (2017) GAP numbers differ from Webster et al. due to parser differences. Please make the conversion script and thresholds explicit and ensure the identical script and parser are applied to all compared models. The "state-of-the-art" claim also requires comparison with other published systems, not only the two Lee et al. models.
minor comments (5)
  1. [§5, footnote 2] The two URLs for the released models are identical (https://lil.cs.washington.edu/coref/final.tgz); one of them likely should point to the Lee et al. (2017) model.
  2. [§4.1] The definitions of M and F overlap for first names with male proportion exactly 0.5; clarify which list such names belong to.
  3. [§3] The notation g_adv_i(g) = ∇_{g_i} L({g_i}) could be misread as a gradient with respect to a single span; specify that the gradient is computed with respect to each span representation while holding the others fixed, and explain why the span embeddings are not normalized before adding the perturbation.
  4. [Table 3 caption] The underlining described in the caption is not visible in the table; please ensure the final typesetting shows which differences are significant and report the exact p-values.
  5. [References] The GAP dataset reference has an incomplete title ("gendered ambiguou"); correct it to the full title of Webster et al. (2018).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the adversarial-training results are empirical comparisons on held-out benchmarks and do not reduce to fitted inputs or self-cited theorems.

full rationale

The paper makes no formal derivation that reduces to its own inputs. Its central claims are empirical: applying FGSM adversarial training to span representations of an existing coreference model and evaluating on the original CoNLL test set, a no-leakage transformed test set, and the GAP test set. The total loss L_total(g) = alpha L(g) + (1-alpha) L_adv(g) is a standard regularized objective, and while L_adv is defined from the model's own loss gradient, that is a definition of the training procedure rather than a predicted quantity derived from fitted values. The hand-chosen constants (alpha = 0.6, epsilon = 1, and 355,000 training iterations) are hyperparameters, and all reported results are on independent test sets rather than on the training data used to choose them. The paper does cite prior work involving its own authors, specifically Agarwal et al. (2018) for entity-type statistics and Khashabi et al. (2016) and Chang et al. (2013) as related or expository references, but none of these citations carries the improvement claim; that claim rests on comparisons with externally released Lee et al. (2017, 2018) models. Footnote 3 discloses a preprocessing error that caused small differences from the ACL Anthology version, which is an evaluation correction and not a circular step. The baseline comparison could be questioned on causal-identification grounds because the released checkpoint is a single draw and was not retrained under the identical protocol, but that is a methodological confound, not an instance of a prediction being equivalent by construction to its input. The paper is self-contained against external benchmarks, so no circularity is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the standard Lee et al. architecture and on the assumption that the name-replacement procedure leaves coreference structure intact. The model-specific hand-picked values are alpha and epsilon; the replacement procedure also uses a hand-chosen gender threshold. No new theoretical entities are postulated.

free parameters (3)
  • alpha = 0.6
    Mixing weight in Ltotal = alpha * L + (1-alpha) * Ladv; selected because it 'work[s] well', with no sensitivity analysis or ablation reported.
  • epsilon = 1
    Scale of the adversarial perturbation applied to span representations; selected by experiments, with no sensitivity analysis reported.
  • male first-name proportion threshold = 0.5
    Threshold used to classify first names as male or female in the PER replacement procedure; hand-chosen without validation or sensitivity analysis.
assumptions (3)
  • domain assumption Replacing PER and GPE names with the described gender and location-type controls preserves the gold coreference structure of each document.
    Section 4 states that for PER/GPE it is easy to change names 'without changing the true coreference structure of the document.' The validity of the no-leakage evaluation depends on this assumption.
  • domain assumption The Lee et al. architecture and its pretrained embeddings, span representations, and hyperparameters are valid and sufficiently well specified to support the comparison.
    The paper builds directly on the Lee et al. (2018) model and uses its released pretrained models and training hyperparameters without independently revalidating them.
  • ad hoc to paper Adversarial perturbations computed on span representations generalize to unseen names and to other datasets.
    This is the central empirical hypothesis tested in the paper. The authors report that perturbing character embeddings was less effective, but no theoretical or independent evidence is provided for why span representations are the right layer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Generalization in Coreference Resolution via Adversarial Training." pith.science (2026). https://pith.science/paper/4PN3GEMG

@misc{pith2026190804728,
  author       = {Pith},
  title        = {Pith review of: Improving Generalization in Coreference Resolution via Adversarial Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4PN3GEMG}},
  note         = {Machine review of arXiv:1908.04728}
}
read the original abstract

In order for coreference resolution systems to be useful in practice, they must be able to generalize to new text. In this work, we demonstrate that the performance of the state-of-the-art system decreases when the names of PER and GPE named entities in the CoNLL dataset are changed to names that do not occur in the training set. We use the technique of adversarial gradient-based training to retrain the state-of-the-art system and demonstrate that the retrained system achieves higher performance on the CoNLL dataset (both with and without the change of named entities) and the GAP dataset.

Figures

Figures reproduced from arXiv: 1908.04728 by the authors.

Figure 1
Figure 1. For each mention, the model computes scores [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 11 canonical work pages

  1. [1]

    Oshin Agarwal, Sanjay Subramanian, Ani Nenkova, and Dan Roth. 2018. Named person coreference in english news. arXiv preprint arXiv:1810.11476

    work page Pith review arXiv 2018

  2. [2]

    Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890--2896

    2018

  3. [3]

    Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Adversarial training for multi-context joint entity and relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2830--2836

    work page 2018

  4. [4]

    Kai-Wei Chang, Rajhans Samdani, and Dan Roth. 2013. http://cogcomp.org/papers/ChangSaRo13.pdf A constrained latent variable model for coreference resolution . In EMNLP

    work page 2013

  5. [5]

    Abbas Ghaddar and Philippe Langlais. 2016. Wikicoref: An english coreference-annotated corpus of wikipedia articles. In LREC

    work page 2016

  6. [6]

    Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. http://arxiv.org/abs/1412.6572 Explaining and harnessing adversarial examples . In International Conference on Learning Representations

    arXiv 2015

  7. [7]

    Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780

    1997

  8. [8]

    Matthew Honnibal and Mark Johnson. 2015. An improved non-monotonic transition system for dependency parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1373--1378

    work page 2015

Show all 24 references
  1. [9]

    Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, and Dan Roth. 2016. http://cogcomp.org/papers/KKSCER16.pdf Question answering via integer programming over semi-structured knowledge . In Proc. of the International Joint Conference on Artificial Intell...

    2016

  2. [10]

    Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188--197

    2017

  3. [11]

    Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Sh...

    2018

  4. [12]

    Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153--157

    1947

  5. [13]

    Dai, and Ian Goodfellow

    Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. 2017. https://arxiv.org/abs/1605.07725 Adversarial training methods for semi-supervised text classification . ICLR

    2017 arXiv

  6. [14]

    Nafise Sadat Moosavi and Michael Strube. 2017. Lexical features in coreference resolution: To be used with caution. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 14--19

    2017

  7. [15]

    Nafise Sadat Moosavi and Michael Strube. 2018. Using linguistic features to improve the generalization capability of neural coreference resolvers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 193--203

    2018

  8. [16]

    Eric W Noreen. 1989. Computer-intensive methods for testing hypotheses. Wiley New York

    1989

  9. [17]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532--1543

    2014

  10. [18]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: H...

    2018

  11. [19]

    Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 1--40. Association for Computational Linguistics

    2012

  12. [20]

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929--1958

    2014

  13. [21]

    Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the gap: A balanced corpus of gendered ambiguou. In Transactions of the ACL, page to appear

    2018

  14. [22]

    Yi Wu, David Bamman, and Stuart Russell. 2017. Adversarial training for relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1778--1783

    2017

  15. [23]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  16. [24]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.