Partially-supervised Mention Detection

James Henderson; Lesly Miculicich

REVIEW 3 major objections 5 minor 19 references

Partially-supervised Mention Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Modifying the loss for unannotated mentions improves mention detection and coreference resolution under partial annotation.

desk verdict Solid empirical study of two simple partial-annotation fixes for mention detection, but the headline recall number is internally inconsistent and the coreference gains are too small to support the 'significant improvement' claim. read the letter →

arxiv 1908.09507 v1 pith:GPSFV7XY submitted 2019-08-26 cs.CL cs.LG

Lesly Miculicich , James Henderson This is my paper · ORCID

classification cs.CLcs.LG

keywords mentiondetectionpartialannotationcoreferenceresolutionmultitasklearningweightedlosssoft-targetclassificationsequencetaggingspanscoring

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tackles mention detection when training data is only partially annotated: for example, coreference corpora mark spans that belong to a coreference chain, leaving other true mentions unlabeled. It proposes two simple loss modifications—weighted loss and soft-target classification—to stop treating unannotated spans as hard negatives, encouraging the model to predict mention-like spans even without labels. These are tested with two neural detectors, a sequence tagger and an exhaustive span scorer, in a multitask system that also trains coreference resolution. The central result is that both modifications increase recall and F1 for mention detection, and the span-scoring model with soft targets raises coreference average F1 from 67.2 to 67.6 over the baseline resolver.

What carries the argument

The central mechanism is a modification of the training objective for negative examples. In the sequence tagger, negative words ('-' label) receive loss weight $w<1$ or a softened target distribution $(\rho,\rho,\rho,1-3\rho)$ over the four tag symbols; in the span scorer, unannotated spans are treated as negatives with weight $w$ or target probability $\rho$. This lowers the cost of predicting an unannotated span, pushing the detector toward higher recall. The span scorer itself scores every possible span, and its mention probability $P(m_{ij})$ enters the coreference resolver as a scaled score $s_m = v\,P(m_{ij})$, so mention-detection uncertainty is integrated into coreference decisions.

What would settle it

Fully annotate all mentions in a held-out portion of the OntoNotes test set, train the soft-target span-scoring model on the partially annotated data, and compare its detections against the full annotations. If the recall gain over the baseline consists mostly of spans that are true mentions, the distributional assumption holds; if the gain is mostly false positives or differs by mention type (e.g., pronouns vs. nominals), the assumption fails.

Watch

Extended reading notes

Core claim

The discovery is that partial annotation can be exploited directly at the loss level, without enumerating all plausible mentions. For each negative example, the loss is either down-weighted by a factor $w$ or assigned a small positive target probability $\rho$; this makes false positives cheaper during training, so the detector learns to output spans the annotator did not label. On CoNLL 2012 OntoNotes, the span-scoring model with $\rho=0.1$ reaches 82.9 mention detection F1 and 67.6 coreference average F1, topping the 67.2 of the end-to-end baseline resolver, while sequence tagging reaches its best coreference F1 of 64.1 with weighted loss $w=0.01$.

Load-bearing premise

The claim rests on the assumption that unannotated mentions in coreference data are roughly the same distribution as annotated ones, so the true mention distribution can be learned by extrapolating from labeled samples.

Editorial extensions

If this is right

Mention detection can be trained from coreference data alone, without syntactic parsing, and still recover many unannotated mentions.
The two loss modifications are architecture-agnostic and improve both a sequence-tagging and an exhaustive span-scoring detector.
Recall gains on mention detection propagate to the downstream coreference task, lifting average F1 above a strong end-to-end baseline.
The best modification differs by architecture: weighted loss helps sequence tagging more, soft targets help span scoring more.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same negative-example softening could help other partially supervised tasks, such as relation extraction where unlabeled relations are abundant.
A direct way to test the paper's distributional assumption is to fully annotate a held-out subset and verify that the recall gains match true missing mentions rather than systematic false positives.
The optimal $\rho$ and $w$ are likely corpus-dependent, so applying this approach to a new domain should include a development-set search paired with an annotation-bias analysis.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Take a look at this one if you work on mention detection or coreference with partially annotated data. The core idea is simple and sensible: when negative spans may actually be unannotated mentions, down-weight their loss (weighted loss) or replace hard zeros with soft targets. The paper applies both to a sequence tagger and a span scorer, with a multitask coreference resolver on top. The methods are standard, but the application is legitimate and the seq2seq tagger for nested mentions is a neat formulation.

What is genuinely useful: the span-scoring variant with soft targets gives the best mention-detection numbers in Table 2 (78.4 recall vs 75.3 baseline), and the coreference F1 improves from 67.0 to 67.6. The improvements are consistent across both models, which suggests there is something real here. The writing is clear and the hyperparameters were tuned on dev.

Soft spots, in order of severity. First, the recall claim in §6.3 does not cohere with Table 2. The text says the weighted-loss sequence tagger reaches 90.5% recall, while Table 2 reports 77.3% for the same configuration. No definition of the recall metric is given for §6.3. If that number counts unannotated-but-mention-like spans as correct, then the gain is partly built into the scorer. If it is a typo or a dev-set number, the paper needs to say so. As written, the reader cannot tell which recall numbers are the real result, and the central claim of the paper rests on this.

Second, the 'significant improvement over the baseline coreference resolver' (Section 1) is not backed by any significance test. The gap is 0.4 F1 (67.6 vs 67.2), which could easily be noise. No error bars are reported anywhere.

Third, the underlying distributional assumption — that unannotated mentions look like annotated ones — is asserted, not tested. The paper would be stronger if it measured precision against a fully annotated sample or at least discussed what happens when that assumption fails.

Fourth, no code is released, so the numbers cannot be independently checked.

I would not call this a breakthrough, but it is a fair empirical study of a real problem. Send it to reviewers, but require the authors to reconcile the 90.5/77.3 discrepancy, add significance testing or at least error bars, and ideally release code. If those issues are fixed, the paper is citable for the application of loss modification to partial mention detection.

Referee Report

3 major / 5 minor

Summary. The paper addresses mention detection under partial annotation, where coreference data contains only mentions that belong to a coreference chain. It proposes two training adjustments for partial annotation, a weighted loss and soft-target classification, and two neural mention detection models, a sequence tagging approach and a span-scoring exhaustive search approach. The methods are evaluated on the English OntoNotes CoNLL-2012 data, with coreference resolution as a downstream task in a multitask learning setup. The central claim is that the loss modifications improve recall and F1 for both mention detection and coreference resolution, and that the span-scoring model with soft targets significantly improves over the Lee et al. (2017) coreference baseline.

Significance. If the empirical results hold, the paper offers simple and potentially general techniques for partially supervised mention detection, and it provides a useful comparison between sequence tagging and exhaustive span scoring within a multitask coreference system. The experimental design has strengths: hyperparameters are tuned on development data, an external baseline is used, and the loss modifications are not derived from the test results, so no circularity is apparent. However, the supporting evidence is currently weakened by an internal inconsistency in the reported recall numbers and by the absence of significance testing, especially for the coreference gains. The central claim is therefore defensible only after these issues are resolved.

major comments (3)

[§6.3 and Table 2] Section 6.3 reports that the sequence tagging model with weighted loss w=0.01 reaches 90.5% recall, while Table 2 lists the same configuration at 77.3% recall in the multitask coreference evaluation, and Table 1 gives the unmodified sequence tagging recall as 73.7% (versus 73.1% in Table 2). Since the recall gain is the paper's main quantitative evidence, this discrepancy must be resolved: if the 90.5% figure counts as correct spans that are unannotated but mention-like, then part of the gain is an artifact of the partial-annotation assumption being built into the scorer; if it is a dev-set number or a typo, the central result is not reliably reported as it stands.
[§6.2 and Table 2] The paper uses the word 'significant' for the improvements (Introduction and §6.2), but no significance tests, confidence intervals, or repeated-run variance are reported. The coreference improvement of the best span-scoring soft-target model over Lee et al. (2017) is 67.6 versus 67.2 F1, which is within the range of run-to-run noise typical for neural coreference systems. The authors should either add significance testing, for example bootstrap over documents or multiple random seeds, or temper the wording of the claims.
[§4 and §1] The load-bearing assumption that unannotated mentions have approximately the same distribution as annotated ones is stated in Section 1 and used in Section 4, but it is never validated. If annotators systematically omit certain mention types, the weighted loss and soft-target modifications may simply increase false positives instead of recovering missing true mentions. A concrete check would be to measure whether the false positives introduced by the modified losses coincide with held-out mention annotations, or to compare the distribution of predicted extra mentions against a manually annotated gold mention set.

minor comments (5)

[§6.2] In the text, 'conference resolution' should be 'coreference resolution'.
[§4] There are typographical errors: 'distiction' should be 'distinction', and 'LSMT' should be 'LSTM'.
[§1 and §6] The dataset name is spelled inconsistently: 'Ontonotes' appears in Section 1 while 'OntoNotes' appears in Section 6; the latter spelling should be used throughout.
[§6.1] The word 'reslting' in the discussion of attention embeddings should be 'resulting'.
[§6] The relation between Table 1 and Table 2 should be stated explicitly: Table 1 reports standalone mention detectors, while Table 2 reports mention detection under the multitask coreference training regime. Without this clarification, the reader cannot tell which evaluation setup the recall figures in Section 6.3 refer to.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claims are empirical, compared against an external baseline, with hyperparameters tuned on development data.

full rationale

The paper makes no derivation claim that reduces to its own inputs. The two loss modifications in Section 4 (weighted loss with weight w, soft targets with probability rho) are training-objective changes; they are not fitted to the target results and are not defined in terms of the reported mention or coreference scores. Section 6 explicitly states that rho, w, and tau are optimized on the development set, and the central coreference comparison is against the external Lee et al. (2017) baseline in Table 2. The self-citations in the introduction (Miculicich Werlen and Popescu-Belis, 2017; Miculicich Werlen et al., 2018; Miculicich et al., 2018) only motivate downstream applications and do not carry the experimental argument. No uniqueness theorem, ansatz, or known result is imported from prior work by the same authors. The distributional assumption about unannotated mentions is an empirical assumption, not a circular one: if it fails the recall gains may not transfer, but that is a correctness risk, not a reduction of the conclusion to the premise. The reviewer should separately note a reproducibility inconsistency: Section 6.3 reports 90.5% recall for sequence tagging with weighted loss w=0.01, while Table 2 reports 77.3% for the same configuration; and the 67.6 vs 67.2 coreference gain is called 'significant' without a significance test. These are internal-consistency issues, not circularity, so they do not change the circularity score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on two free hyperparameters (w, rho) tuned on dev, a threshold tau, and the domain assumption that missing mentions share the annotation distribution. No new entities are introduced.

free parameters (3)

w (negative loss weight) = 0.01 for sequence tagging; 0.3 for span scoring
Weight applied to negative examples in the loss to favor recall; chosen by development set performance, reported in Table 2.
rho (soft-target probability) = 0.1 for both models
Probability assigned to negative examples in soft-target classification; tuned on dev set, reported in Table 2.
tau (detection threshold) = tau=0.5 in Table 1; final tau values from dev used in Table 2 and Figure 3
Threshold for predicting spans as mentions in span scoring; optimized on development set, exact values for Table 2 not given.

assumptions (3)

domain assumption Unannotated mentions have approximately the same distribution as annotated ones
Sections 1 and 4 state this assumption is needed to learn the true mention distribution from partially annotated coreference data.
domain assumption The LSTM-based tagger captures the simple grammar of the tag labels with very few mistakes
Section 4 uses this to justify treating word-level decisions as valid negative examples in sequence tagging loss, despite potential inconsistencies like overlapping or non-ending mentions.
domain assumption Coreference data provides a suitable partial annotation for mention detection
Section 1 states coreference annotations contain only mentions that belong to a chain, which is the basis for the task formulation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Partially-supervised Mention Detection." pith.science (2026). https://pith.science/paper/GPSFV7XY

@misc{pith2026190809507,
  author       = {Pith},
  title        = {Pith review of: Partially-supervised Mention Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GPSFV7XY}},
  note         = {Machine review of arXiv:1908.09507}
}

read the original abstract

Learning to detect entity mentions without using syntactic information can be useful for integration and joint optimization with other tasks. However, it is common to have partially annotated data for this problem. Here, we investigate two approaches to deal with partial annotation of mentions: weighted loss and soft-target classification. We also propose two neural mention detection approaches: a sequence tagging, and an exhaustive search. We evaluate our methods with coreference resolution as a downstream task, using multitask learning. The results show that the recall and F1 score improve for all methods.

Figures

Figures reproduced from arXiv: 1908.09507 by the authors.

**Figure 1.** Samples from CoNLL 2012. Annotated mentions are within brackets contains only mentions that belong to a coreference chain, not single entity-mentions. Nevertheless, the missing mentions have approximately the same distribution as the annotated ones, so we can still learn this distribution from the data [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Tagged sentence example (h1, ..., hM) = BiLSTM(X) (4) dt = LSTM(y1, ..., yt−1) (5) where the decoder is initialized with the last states of the bidirectional encoder, d0 = hM. At decoding time, we use a beam search approach to obtain the sequence. The complexity of the model is linear with respect to the number of words. It can be parallelized at training time given that it uses ground-truth data for the conditione… view at source ↗

**Figure 3.** Recall comparison tagging, the weighted loss results in higher performance, while for the span scoring soft-targets work best. In both cases, the recall increases with a small decrease in precision, which improves the F1 score of mention detection, and in turn improves conference resolution. 6.3 Recall performance [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 7 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Roberto Cipolla, Yarin Gal, and Alex Kendall. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7482--7491. IEEE

work page 2018
[4]

Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. 2018. https://doi.org/10.18653/v1/N18-1204 Neural text generation in stories using entity representations as context . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 2250--2260...

doi:10.18653/v1/n18-1204 2018
[5]

R Florian, H Hassan, A Ittycheriah, H Jing, N Kambhatla, X Luo, N Nicolov, and S Roukos. 2004. https://www.aclweb.org/anthology/N04-1001 A statistical model for multilingual entity detection and tracking . In HLT-NAACL 2004: Main Proceedings, pages 1--8, Boston, Massachusetts, USA. Association for Computational Linguistics

work page 2004
[6]

Alex Graves and J \"u rgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5-6):602--610

2005
[7]

Sepp Hochreiter and J \"u urgen Schmidhuber. 1997. Long-short term memory. Neural Computation, 9(8):1735--1780

work page 1997
[8]

Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. https://doi.org/10.18653/v1/N18-1131 A neural layered model for nested named entity recognition . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1446--1459, New Orleans, Louis...

doi:10.18653/v1/n18-1131 2018

Show all 19 references

[9]

Arzoo Katiyar and Claire Cardie. 2018. https://doi.org/10.18653/v1/N18-1079 Nested named entity recognition revisited . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (...

2018 doi
[10]

Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. https://doi.org/10.18653/v1/N16-1030 Neural architectures for named entity recognition . In Proceedings of the 2016 Conference of the North American Chapter of the Association for...

2016 doi
[11]

Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/D17-1018 End-to-end neural coreference resolution . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188--197, Copenhagen, Denmark. Associ...

2017 doi
[12]

Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. https://doi.org/10.18653/v1/D18-1325 Document-level neural machine translation with hierarchical attention networks . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro...

2018 doi
[13]

Lesly Miculicich Werlen, Nikolaos Pappas, Dhananjay Ram, and Andrei Popescu-Belis. 2018. https://doi.org/10.18653/v1/N18-1124 Self-attentive residual decoder for neural machine translation . In Proceedings of the 2018 Conference of the North A merican Chapter of the Associatio...

2018 doi
[14]

Lesly Miculicich Werlen and Andrei Popescu-Belis. 2017. https://doi.org/10.18653/v1/W17-1505 Using coreference links to improve S panish-to- E nglish machine translation . In Proceedings of the 2nd Workshop on Coreference Resolution Beyond O nto N otes ( CORBON 2017) , pages 3...

2017 doi
[15]

Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. https://www.aclweb.org/anthology/P09-1113 Distant supervision for relation extraction without labeled data . In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International J...

2009
[16]

Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. https://www.aclweb.org/anthology/W12-4501 Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes . In Joint Conference on EMNLP and CoNLL - Shared Task, pages...

2012
[17]

Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. https://doi.org/10.1162/089120101753342653 A machine learning approach to coreference resolution of noun phrases . Computational Linguistics, 27(4):521--544

2001 doi
[18]

Bailin Wang, Wei Lu, Yu Wang, and Hongxia Jin. 2018. https://www.aclweb.org/anthology/D18-1124 A neural transition-based model for nested mention recognition . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1011--1017, Brussels...

2018
[19]

Mingbin Xu, Hui Jiang, and Sedtawut Watcharawittayakul. 2017. https://doi.org/10.18653/v1/P17-1114 A local detection approach for named entity recognition and mention detection . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume...

2017 doi

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Roberto Cipolla, Yarin Gal, and Alex Kendall. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7482--7491. IEEE

work page 2018

[4] [4]

Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. 2018. https://doi.org/10.18653/v1/N18-1204 Neural text generation in stories using entity representations as context . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 2250--2260...

doi:10.18653/v1/n18-1204 2018

[5] [5]

R Florian, H Hassan, A Ittycheriah, H Jing, N Kambhatla, X Luo, N Nicolov, and S Roukos. 2004. https://www.aclweb.org/anthology/N04-1001 A statistical model for multilingual entity detection and tracking . In HLT-NAACL 2004: Main Proceedings, pages 1--8, Boston, Massachusetts, USA. Association for Computational Linguistics

work page 2004

[6] [6]

Alex Graves and J \"u rgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5-6):602--610

2005

[7] [7]

Sepp Hochreiter and J \"u urgen Schmidhuber. 1997. Long-short term memory. Neural Computation, 9(8):1735--1780

work page 1997

[8] [8]

Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. https://doi.org/10.18653/v1/N18-1131 A neural layered model for nested named entity recognition . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1446--1459, New Orleans, Louis...

doi:10.18653/v1/n18-1131 2018