{"id":"6b9145aa-7153-47a8-808f-79da6f82d0c0","arxiv_id":"1908.04728","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adversarial training on span representations slightly but generally improves coreference resolution accuracy, including on inputs whose named entities do not appear in training.","lead":"Coreference models lose accuracy when test documents contain person or place names never seen in training. The authors show that training with gradient-based adversarial noise on the model's internal span representations recovers part of that loss and improves accuracy on two benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal claim that adversarial training improves coreference is confounded by the baseline: the comparison uses an off-the-shelf pretrained checkpoint, not a baseline retrained under the same protocol.","rationale":"The central claim requires that the adversarial loss, not a favorable training run, explains the gains. Using a released pretrained checkpoint for the baseline breaks the causal comparison because the two conditions differ in training run as well as loss. This concern is more load-bearing than the reader's no-leakage concern: even if the no-leakage set is perfectly constructed, the pretrained-baseline confound would leave Tables 3 and 4 unable to support the attribution. The paper does provide some independent support (the method is simple, and a code link is given; the GAP improvement is statistically significant by McNemar), so the appropriate outcome is not rejection but a conditional request to retrain the baseline across seeds. The reader's weakest_assumption identified a different issue, so I disagree on that point, but the reader's overall CONDITIONAL verdict remains appropriate.","tokens_in":7130,"tokens_out":13078,"duration_ms":132168,"concrete_test":"Retrain the non-adversarial Lee et al. (2018) baseline with the exact same code, training hyperparameters, number of iterations, and no-leakage preprocessing as used for the +Adv. Training model, for at least 5 random seeds, and run the adversarial model for the same 5 seeds. Compare paired CoNLL F1 on the original and no-leakage test sets and GAP F1 with mean, standard deviation, and a paired significance test. If the baseline distribution overlaps or exceeds the adversarial distribution, the improvement cannot be attributed to adversarial training.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 says the +Adv. Training model is trained in-house for 355,000 iterations, but the Lee et al. (2018) baseline is the released pretrained checkpoint. A single released checkpoint is one draw from a stochastic training process; it does not establish how the same architecture trained without the adversarial loss would perform in this pipeline. The reported margins are small (CoNLL: 73.23 vs. 72.96 and 72.32 vs. 71.84; GAP: 74.7 vs. 73.3), so random initialization, training order, and hardware nondeterminism are all the same magnitude as the claimed effect. This affects every reported result, including the GAP comparison and the no-leakage result, not just one evaluation. Without retraining the non-adversarial baseline under the identical code, hyperparameters, data order, and seeds, the headline 'state-of-the-art' claim is not causally identified.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies how coreference resolution generalizes to unseen names. The authors construct a \"No Leakage\" version of the CoNLL-2012 test set by replacing PER and GPE named entities with names not seen in training, controlling for gender and location category. They apply FGSM adversarial training to the span representations of the Lee et al. (2018) end-to-end coreference model, minimizing alpha*L + (1-alpha)*L_adv with alpha=0.6 and epsilon=1. They report that the Lee et al. system drops by more than 1 F1 under the No Leakage modification (71.84 vs 72.96) and that the adversarially trained model improves over the original model on the original CoNLL test set (73.23 vs 72.96), on the No Leakage set (72.32 vs 71.84), and on the GAP dataset (74.7 vs 73.3 overall F1), concluding that the adversarially trained system is a new state of the art.","tokens_in":7323,"tokens_out":6225,"duration_ms":58137,"significance":"If the results hold, this paper would demonstrate a simple and computationally cheap regularization that improves coreference generalization and transfers across datasets, and the No Leakage diagnostic is a useful contribution. The paper ships a code link and uses public benchmarks; the GAP improvements are statistically significant (McNemar p<0.05) and directionally consistent across gender. However, the central causal claim is not fully supported because the comparison baseline is a released checkpoint rather than a retrained control, the CoNLL margins are small and not significant at conventional levels, and the No Leakage assumption that replacements preserve gold structure is unvalidated. These issues are addressable with additional experiments.","major_comments":[{"comment":"The central comparison is confounded by the baseline protocol. The adversarially trained model is trained in-house for 355,000 iterations, while the Lee et al. (2018) baseline is the released pretrained checkpoint (Section 5). A single released checkpoint is one draw from a stochastic training process, and the reported margins (0.27 and 0.48 CoNLL F1; 1.4 GAP F1) are of the same magnitude as run-to-run variation from initialization, data order, and hardware nondeterminism. To support the claim that the adversarial loss causes the improvement, the authors must retrain the non-adversarial Lee et al. model under identical code, hyperparameters, data ordering, and multiple seeds, and report the mean and spread. This affects every reported result, including the GAP claim.","section":"§5, Tables 3 and 4"},{"comment":"The CoNLL improvements are not established statistically. The caption uses p<0.21 as the significance threshold for underlining, which is far weaker than the conventional p<0.05, and no exact p-values or confidence intervals are reported. With differences of 0.27 and 0.48 F1, the \"consistently performs better\" claim for CoNLL is unsupported. Please report exact p-values, confidence intervals (or bootstrap), and ideally multiple runs.","section":"§5.1, Table 3"},{"comment":"The No Leakage evaluation assumes that replacing PER and GPE names does not change true coreference structure, but this assumption is not validated. Gender and location-type controls do not account for all ways in which a name change can alter a document's coreference (e.g., nominal predicates, titles, generic or metonymic uses, number/gender cues beyond pronouns, and parser head errors). If replacements introduce artifacts, the measured degradation and the apparent benefit of adversarial training could be evaluation artifacts. Please validate the replacement protocol, for example by human annotation of a sample of modified documents, and report an error analysis.","section":"§4, §4.1, §4.2"},{"comment":"Hyperparameter selection is underreported. alpha=0.6 and epsilon=1 are said to \"work well\" without a search criterion or sensitivity analysis, and the claim that perturbing character embeddings \"was not as effective\" is not accompanied by numbers. Because the central claim depends on the adversarial objective, please provide a small sensitivity analysis and, crucially, an ablation that trains the same architecture without the adversarial term under the identical protocol (this also addresses the baseline issue).","section":"§3"},{"comment":"The GAP comparison may not be fully controlled. The system outputs must be converted to binary (pronoun, name) predictions, and the authors note that their Lee et al. (2017) GAP numbers differ from Webster et al. due to parser differences. Please make the conversion script and thresholds explicit and ensure the identical script and parser are applied to all compared models. The \"state-of-the-art\" claim also requires comparison with other published systems, not only the two Lee et al. models.","section":"§5.2, Table 4"}],"minor_comments":[{"comment":"The two URLs for the released models are identical (https://lil.cs.washington.edu/coref/final.tgz); one of them likely should point to the Lee et al. (2017) model.","section":"§5, footnote 2"},{"comment":"The definitions of M and F overlap for first names with male proportion exactly 0.5; clarify which list such names belong to.","section":"§4.1"},{"comment":"The notation g_adv_i(g) = ∇_{g_i} L({g_i}) could be misread as a gradient with respect to a single span; specify that the gradient is computed with respect to each span representation while holding the others fixed, and explain why the span embeddings are not normalized before adding the perturbation.","section":"§3"},{"comment":"The underlining described in the caption is not visible in the table; please ensure the final typesetting shows which differences are significant and report the exact p-values.","section":"Table 3 caption"},{"comment":"The GAP dataset reference has an incomplete title (\"gendered ambiguou\"); correct it to the full title of Webster et al. (2018).","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper has a genuinely reusable evaluation idea, but its central claim about adversarial training isn't as established as the abstract makes it sound.\n\nWhat's new is the no-leakage CoNLL test set. The authors replace PER and GPE names with unseen names while controlling for gender and location category, remove all training-set names from the replacement pools, and use dependency parsing plus gold NER to preserve the coreference structure. That's careful, and it gives the field a concrete stress-test protocol. The observation that the Lee et al. (2018) system loses about 1 F1 when names are replaced is real and doesn't depend on the adversarial comparison.\n\nThe adversarial part is a reasonable extension — applying FGSM to span representations rather than input embeddings. The GAP improvement is statistically significant at p<0.05 and consistent across male and female pronouns, which is a point in the paper's favor.\n\nBut the stress-test concern is correct: Section 5 compares against the released pretrained checkpoint of Lee et al. (2018), not against a baseline retrained under the identical code, hyperparameters, data order, and seeds. With CoNLL margins of 0.27 and 0.48 F1, a single checkpoint is one draw from a stochastic training process; the claimed effect is the same size as random seed noise. The p<0.21 threshold for CoNLL significance is unorthodox and effectively means not significant under any conventional standard. Hyperparameters alpha=0.6 and epsilon=1 appear hand-picked, with no ablation reported. The no-leakage replacement is mostly sound, but subtle distribution shift beyond name novelty is possible; the controls are reasonable, so I wouldn't call it fatal.\n\nThis paper is worth a serious referee, but it needs revision. The authors should retrain the non-adversarial baseline under the same protocol, run multiple seeds, and report confidence intervals. If they can do that, the claim about adversarial training becomes credible. As is, the contribution is the evaluation protocol, not the demonstrated benefit of the method.\n\nFor a reading group, it could spark discussion about evaluation rigor in coreference, but it's not a slam dunk. I'd cite the no-leakage protocol if I were working on coreference robustness, but not the adversarial result without the retrained baseline.","headline":"A useful no-leakage evaluation protocol, but the claimed benefit of adversarial training is not established because the comparison baseline is a single off-the-shelf checkpoint.","tokens_in":7827,"tokens_out":2899,"would_cite":true,"duration_ms":30631,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adversarial training on span representations improves coreference resolution on original, renamed, and GAP test sets.","keywords":["coreference resolution","adversarial training","fast gradient sign method","generalization","no-leakage evaluation","named entity replacement","GAP dataset"],"falsifier":"Re-run the no-leakage evaluation many times with fresh random replacement names drawn from the same or independent name lists and check whether the baseline's drop and the adversarially trained model's gain persist across draws. If ranking flips or the gap shrinks to noise under re-sampling, the reported advantage is an artifact of one particular replacement sample rather than a property of adversarial training.","tokens_in":6925,"feed_emoji":"📈","tokens_out":9445,"duration_ms":83869,"temperature":0.7,"pith_summary":"This paper asks whether the leading neural coreference resolver generalizes beyond the names it saw in training, and reports that it does not: replacing every person and geopolitical name in the CoNLL test set by a name absent from training lowers CoNLL F1 by more than one point. The authors then retrain the same architecture with adversarial gradient training applied to its span representations, and report higher F1 on the unchanged CoNLL test set (73.23 vs 72.96), on the no-leakage test set (72.32 vs 71.84), and on the GAP pronoun dataset (74.7 vs 73.3). The point of the paper is that a generic robustness technique can push an already-strong coreference system to generalize to unseen names, and that perturbing intermediate span representations, rather than input embeddings, is the effective place to do it.","feed_headline":"Adversarial training beats name leakage in coreference","feed_subtitle":"Retraining the top system with gradient perturbations raises CoNLL and GAP scores, even after names are swapped.","key_machinery":"The load-bearing mechanism is adversarial training at the span-representation layer. A span representation is the model's vector for a candidate mention: the concatenation of the contextualized vectors at the span's start and end, an attention-weighted sum of its word vectors, and a learned width embedding, so that the model's predictions and loss depend on the input text only through these vectors. The paper computes the gradient of the coreference loss with respect to each span representation, creates a perturbed span vector by adding a small normalized step $\\epsilon g_i/\\|g_i\\|$ in the gradient direction, and trains on $\\mathcal{L}_{\\mathrm{total}} = \\alpha \\mathcal{L} + (1-\\alpha)\\mathcal{L}_{\\mathrm{adv}}$ with $\\alpha = 0.6$ and $\\epsilon = 1$. The second piece of machinery is the evaluation protocol itself: a no-leakage test set built by replacing PER names with gender-matched names from census and Wikipedia gazetteer lists and GPE names with GeoNames names of the same location category, removing every name seen in training.","core_discovery":"On its own terms, the paper establishes two claims. First, the current best CoNLL-trained system is vulnerable to name leakage: when PER and GPE mentions are replaced by gender-matched or location-category-matched names that never occur in training, performance on the CoNLL test set drops by more than 1 F1. Second, retraining that exact system with the fast-gradient-sign method applied at the span-representation layer fixes most of this vulnerability and improves the original test score as well, yielding 73.23 vs 72.96 on the original CoNLL set, 72.32 vs 71.84 on the no-leakage set, and 74.7 vs 73.3 on GAP. The paper therefore claims a new state of the art on all three test sets. The key is not a new architecture but a changed training objective on the same architecture.","pith_inferences":["A testable extension the paper leaves implicit is an ablation over $\\alpha$ and $\\epsilon$; a narrow band of useful perturbation sizes would suggest margin hardening, while a flat response would suggest the gain is mostly extra noise.","The no-leakage protocol could become a standard 'unseen entity' evaluation for span-based models, since it changes one axis of variation while keeping genre and annotation fixed.","If the perturbation is truly acting on span representations, the same training objective should improve robustness on other span-labeling problems, such as named entity recognition or relation extraction, and that is a direct empirical check of the mechanism.","Because the reported numbers come from a single random draw of replacement names, the paper's point estimates should be read as one sample; averaging over multiple draws would give a more stable estimate of the method's advantage."],"forward_implications":["Retraining with the span-level adversarial objective raises F1 on the unchanged CoNLL test set, so the method acts as a regularizer rather than only a defense against renamed inputs.","The larger relative gain on the no-leakage set indicates the trained system relies less on memorized PER and GPE names and more on structural cues like gender and location category.","The GAP improvement shows the benefit transfers to Wikipedia text and to a pronoun-to-name binary decision task, not just to the CoNLL genre mix.","The adversarial step leaves the GAP male/female F1 ratio essentially unchanged, so the gain does not come at the cost of increased gender bias, and it does not reduce that bias either."],"supporting_citations":[{"why":"Supplies the state-of-the-art coreference architecture and pretrained model that the paper retrains and compares against.","marker":"(Lee et al., 2018)"},{"why":"Supplies the fast-gradient-sign-method adversarial training recipe, adapted here to span representations.","marker":"(Miyato et al., 2017)"},{"why":"Introduces FGSM and the adversarial example intuition the method builds on.","marker":"(Goodfellow et al., 2015)"},{"why":"Defines the CoNLL-2012 dataset and evaluation metric used for the main experiments.","marker":"(Pradhan et al., 2012)"},{"why":"Provides the GAP dataset and its F1 scoring used for the out-of-domain evaluation.","marker":"(Webster et al., 2018)"},{"why":"Provides the prior demonstration of generalization failure in neural coreference that motivates this work.","marker":"(Moosavi and Strube, 2018)"},{"why":"Supplies the spaCy dependency parser used to identify mention heads for the name-replacement protocol.","marker":"(Honnibal and Johnson, 2015)"},{"why":"Contributes the span representation construction and is the earlier baseline compared on GAP.","marker":"(Lee et al., 2017)"}],"fun_headline_variants":["Adversarial retraining plugs coreference name leak","Beat name leakage with adversarial training","Name swap resilience from adversarial training","Coreference scores climb after adversarial retrain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that swapping a person's name for another gender-matched name and a place for a same-category place never changes the gold coreference structure, so any measured drop is caused by unfamiliar names rather than by artifacts of the replacement.","fun_headline_variants_meta":{"raw":{"variants":["Adversarial retraining plugs coreference name leak","Beat name leakage with adversarial training","Name swap resilience from adversarial training","Coreference scores climb after adversarial retrain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000921,"raw_usage":{"total_tokens":3885,"prompt_tokens":813,"completion_tokens":3072,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":3019}},"tokens_in":429,"tokens_out":3072,"duration_ms":20042,"temperature":1.0,"reasoning_tokens":3019,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:33:36.790979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the no-leakage evaluation many times with fresh random replacement names drawn from the same or independent name lists and check whether the baseline's drop and the adversarially trained model's gain persist across draws. If ranking flips or the gap shrinks to noise under re-sampling, the reported advantage is an artifact of one particular replacement sample rather than a property of adversarial training.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CoNLL-2012 dataset and evaluation metric used for the main experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GAP dataset and its F1 scoring used for the out-of-domain evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the prior demonstration of generalization failure in neural coreference that motivates this work."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the spaCy dependency parser used to identify mention heads for the name-replacement protocol."}],"review_version":1}