{"id":"495ad9ba-108b-4d8a-9f8a-8568f42ec116","arxiv_id":"1908.09507","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Weighted loss and soft-target training improve partially-supervised mention detection and coreference resolution on CoNLL 2012.","lead":"This paper studies how to detect entity mentions in text when the training data only labels some mentions, as in coreference datasets. It shows that two simple tricks, reducing the loss for negative examples or using soft labels, improve recall and F1 for mention detection and a downstream coreference task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central recall claim rests on an internal inconsistency: §6.3 reports 90.5% sequence-tagging recall with weighted loss w=0.01, while Table 2 gives 77.3% for the same configuration.","rationale":"I read the paper as an empirical study whose central claim is that weighted-loss and soft-target modifications improve mention-detection recall and F1, and that these gains carry over to coreference. The methods are described clearly and the core idea is plausible, but the paper's own reported numbers must be consistent for the claim to be verifiable. The reader's weakest assumption was the distributional assumption that unannotated mentions resemble annotated ones; that is a real concern, but it is secondary to the evaluation inconsistency in §6.3. Even if the distributional assumption holds, the central recall result is ambiguous because one configuration is reported as 90.5% in the text and 77.3% in Table 2. Since no code is released and no significance tests are provided, this unresolved inconsistency prevents the reader from assigning a reliable verdict. I therefore agree only partially with the reader's diagnosis: the distributional assumption is not the most load-bearing weak point; the missing common evaluation protocol is. If the authors reconcile the two numbers and confirm that the official CoNLL scorer produces the reported improvements, the paper could be reconsidered as conditional, but as written the central claim is unverdictable.","tokens_in":6128,"tokens_out":6586,"duration_ms":65544,"concrete_test":"Recompute the sequence-tagging model with w=0.01 using the official CoNLL-2012 mention detection scorer and the same multitask training setup as Table 2. If the resulting recall is 77.3, then §6.3's 90.5 must be explained or removed; if it is 90.5, then Table 2's number is wrong. In either case, the central claim requires a single consistent evaluation protocol. Additionally, requesting the Figure 3 scoring script would settle whether unannotated spans were counted as true positives.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The paper's central claim is that the proposed loss modifications reliably improve recall and F1 for mention detection and, downstream, coreference. The most load-bearing evidence for that claim is the recall comparison in §6.3, but the numbers there do not cohere with the paper's own tables. §6.3 states: 'The unmodified sequence tagging model achieves 73.7% recall, and by introducing a weighted loss at w=0.01, it reaches 90.5%.' Table 2 lists the same sequence-tagging configuration with weighted loss w=0.01 at mention Recall=77.3 under the multitask coreference evaluation. Table 1 gives the unmodified sequence-tagging recall as 73.7%, so §6.3 uses the same starting number but a different, unexplained recall definition for the modified model. If the 90.5% figure counts unannotated-but-mention-like spans as correct, then the recall gain is partly an artifact of the partial-annotation assumption being built into the scorer. If it is a typo or a dev-set number, the central quantitative claim is unreliable as reported. The coreference gain behind the 'significant improvement' claim is also tiny (67.6 vs 67.2) and lacks a significance test, but the §6.3 inconsistency is the more basic problem: the reader cannot tell which recall numbers are the real result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses mention detection under partial annotation, where coreference data contains only mentions that belong to a coreference chain. It proposes two training adjustments for partial annotation, a weighted loss and soft-target classification, and two neural mention detection models, a sequence tagging approach and a span-scoring exhaustive search approach. The methods are evaluated on the English OntoNotes CoNLL-2012 data, with coreference resolution as a downstream task in a multitask learning setup. The central claim is that the loss modifications improve recall and F1 for both mention detection and coreference resolution, and that the span-scoring model with soft targets significantly improves over the Lee et al. (2017) coreference baseline.","tokens_in":6438,"tokens_out":5136,"duration_ms":46699,"significance":"If the empirical results hold, the paper offers simple and potentially general techniques for partially supervised mention detection, and it provides a useful comparison between sequence tagging and exhaustive span scoring within a multitask coreference system. The experimental design has strengths: hyperparameters are tuned on development data, an external baseline is used, and the loss modifications are not derived from the test results, so no circularity is apparent. However, the supporting evidence is currently weakened by an internal inconsistency in the reported recall numbers and by the absence of significance testing, especially for the coreference gains. The central claim is therefore defensible only after these issues are resolved.","major_comments":[{"comment":"Section 6.3 reports that the sequence tagging model with weighted loss w=0.01 reaches 90.5% recall, while Table 2 lists the same configuration at 77.3% recall in the multitask coreference evaluation, and Table 1 gives the unmodified sequence tagging recall as 73.7% (versus 73.1% in Table 2). Since the recall gain is the paper's main quantitative evidence, this discrepancy must be resolved: if the 90.5% figure counts as correct spans that are unannotated but mention-like, then part of the gain is an artifact of the partial-annotation assumption being built into the scorer; if it is a dev-set number or a typo, the central result is not reliably reported as it stands.","section":"§6.3 and Table 2"},{"comment":"The paper uses the word 'significant' for the improvements (Introduction and §6.2), but no significance tests, confidence intervals, or repeated-run variance are reported. The coreference improvement of the best span-scoring soft-target model over Lee et al. (2017) is 67.6 versus 67.2 F1, which is within the range of run-to-run noise typical for neural coreference systems. The authors should either add significance testing, for example bootstrap over documents or multiple random seeds, or temper the wording of the claims.","section":"§6.2 and Table 2"},{"comment":"The load-bearing assumption that unannotated mentions have approximately the same distribution as annotated ones is stated in Section 1 and used in Section 4, but it is never validated. If annotators systematically omit certain mention types, the weighted loss and soft-target modifications may simply increase false positives instead of recovering missing true mentions. A concrete check would be to measure whether the false positives introduced by the modified losses coincide with held-out mention annotations, or to compare the distribution of predicted extra mentions against a manually annotated gold mention set.","section":"§4 and §1"}],"minor_comments":[{"comment":"In the text, 'conference resolution' should be 'coreference resolution'.","section":"§6.2"},{"comment":"There are typographical errors: 'distiction' should be 'distinction', and 'LSMT' should be 'LSTM'.","section":"§4"},{"comment":"The dataset name is spelled inconsistently: 'Ontonotes' appears in Section 1 while 'OntoNotes' appears in Section 6; the latter spelling should be used throughout.","section":"§1 and §6"},{"comment":"The word 'reslting' in the discussion of attention embeddings should be 'resulting'.","section":"§6.1"},{"comment":"The relation between Table 1 and Table 2 should be stated explicitly: Table 1 reports standalone mention detectors, while Table 2 reports mention detection under the multitask coreference training regime. Without this clarification, the reader cannot tell which evaluation setup the recall figures in Section 6.3 refer to.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript is a short empirical paper with a potentially useful idea, but the internal inconsistency between §6.3 and Table 2 is the main obstacle. If the authors cannot provide a clear explanation and corrected numbers, I would move toward rejection. Adding significance testing and a reproducibility note about random seeds and number of runs would also be necessary for a journal-level assessment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take a look at this one if you work on mention detection or coreference with partially annotated data. The core idea is simple and sensible: when negative spans may actually be unannotated mentions, down-weight their loss (weighted loss) or replace hard zeros with soft targets. The paper applies both to a sequence tagger and a span scorer, with a multitask coreference resolver on top. The methods are standard, but the application is legitimate and the seq2seq tagger for nested mentions is a neat formulation.\n\nWhat is genuinely useful: the span-scoring variant with soft targets gives the best mention-detection numbers in Table 2 (78.4 recall vs 75.3 baseline), and the coreference F1 improves from 67.0 to 67.6. The improvements are consistent across both models, which suggests there is something real here. The writing is clear and the hyperparameters were tuned on dev.\n\nSoft spots, in order of severity. First, the recall claim in §6.3 does not cohere with Table 2. The text says the weighted-loss sequence tagger reaches 90.5% recall, while Table 2 reports 77.3% for the same configuration. No definition of the recall metric is given for §6.3. If that number counts unannotated-but-mention-like spans as correct, then the gain is partly built into the scorer. If it is a typo or a dev-set number, the paper needs to say so. As written, the reader cannot tell which recall numbers are the real result, and the central claim of the paper rests on this.\n\nSecond, the 'significant improvement over the baseline coreference resolver' (Section 1) is not backed by any significance test. The gap is 0.4 F1 (67.6 vs 67.2), which could easily be noise. No error bars are reported anywhere.\n\nThird, the underlying distributional assumption — that unannotated mentions look like annotated ones — is asserted, not tested. The paper would be stronger if it measured precision against a fully annotated sample or at least discussed what happens when that assumption fails.\n\nFourth, no code is released, so the numbers cannot be independently checked.\n\nI would not call this a breakthrough, but it is a fair empirical study of a real problem. Send it to reviewers, but require the authors to reconcile the 90.5/77.3 discrepancy, add significance testing or at least error bars, and ideally release code. If those issues are fixed, the paper is citable for the application of loss modification to partial mention detection.","headline":"Solid empirical study of two simple partial-annotation fixes for mention detection, but the headline recall number is internally inconsistent and the coreference gains are too small to support the 'significant improvement' claim.","tokens_in":6923,"tokens_out":3269,"would_cite":false,"duration_ms":30286,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Modifying the loss for unannotated mentions improves mention detection and coreference resolution under partial annotation.","keywords":["mention detection","partial annotation","coreference resolution","multitask learning","weighted loss","soft-target classification","sequence tagging","span scoring"],"falsifier":"Fully annotate all mentions in a held-out portion of the OntoNotes test set, train the soft-target span-scoring model on the partially annotated data, and compare its detections against the full annotations. If the recall gain over the baseline consists mostly of spans that are true mentions, the distributional assumption holds; if the gain is mostly false positives or differs by mention type (e.g., pronouns vs. nominals), the assumption fails.","tokens_in":5954,"feed_emoji":"🎯","tokens_out":6621,"duration_ms":57783,"temperature":0.7,"pith_summary":"The paper tackles mention detection when training data is only partially annotated: for example, coreference corpora mark spans that belong to a coreference chain, leaving other true mentions unlabeled. It proposes two simple loss modifications—weighted loss and soft-target classification—to stop treating unannotated spans as hard negatives, encouraging the model to predict mention-like spans even without labels. These are tested with two neural detectors, a sequence tagger and an exhaustive span scorer, in a multitask system that also trains coreference resolution. The central result is that both modifications increase recall and F1 for mention detection, and the span-scoring model with soft targets raises coreference average F1 from 67.2 to 67.6 over the baseline resolver.","feed_headline":"Soft targets lift coreference F1 past baseline","feed_subtitle":"By softening how unlabeled mentions count in the loss, a span-scoring model improves coreference from 67.2 to 67.6 average F1.","key_machinery":"The central mechanism is a modification of the training objective for negative examples. In the sequence tagger, negative words ('-' label) receive loss weight $w<1$ or a softened target distribution $(\\rho,\\rho,\\rho,1-3\\rho)$ over the four tag symbols; in the span scorer, unannotated spans are treated as negatives with weight $w$ or target probability $\\rho$. This lowers the cost of predicting an unannotated span, pushing the detector toward higher recall. The span scorer itself scores every possible span, and its mention probability $P(m_{ij})$ enters the coreference resolver as a scaled score $s_m = v\\,P(m_{ij})$, so mention-detection uncertainty is integrated into coreference decisions.","core_discovery":"The discovery is that partial annotation can be exploited directly at the loss level, without enumerating all plausible mentions. For each negative example, the loss is either down-weighted by a factor $w$ or assigned a small positive target probability $\\rho$; this makes false positives cheaper during training, so the detector learns to output spans the annotator did not label. On CoNLL 2012 OntoNotes, the span-scoring model with $\\rho=0.1$ reaches 82.9 mention detection F1 and 67.6 coreference average F1, topping the 67.2 of the end-to-end baseline resolver, while sequence tagging reaches its best coreference F1 of 64.1 with weighted loss $w=0.01$.","pith_inferences":["The same negative-example softening could help other partially supervised tasks, such as relation extraction where unlabeled relations are abundant.","A direct way to test the paper's distributional assumption is to fully annotate a held-out subset and verify that the recall gains match true missing mentions rather than systematic false positives.","The optimal $\\rho$ and $w$ are likely corpus-dependent, so applying this approach to a new domain should include a development-set search paired with an annotation-bias analysis."],"forward_implications":["Mention detection can be trained from coreference data alone, without syntactic parsing, and still recover many unannotated mentions.","The two loss modifications are architecture-agnostic and improve both a sequence-tagging and an exhaustive span-scoring detector.","Recall gains on mention detection propagate to the downstream coreference task, lifting average F1 above a strong end-to-end baseline.","The best modification differs by architecture: weighted loss helps sequence tagging more, soft targets help span scoring more."],"supporting_citations":[{"why":"Supplies the end-to-end coreference resolver used as the baseline and the span-scoring architecture whose loss is modified.","marker":"Lee et al. (2017)"},{"why":"Provides the CoNLL 2012 OntoNotes dataset and the average F1 metric used for evaluation.","marker":"Pradhan et al. (2012)"},{"why":"Provides the uncertainty-based method for weighting the mention-detection and coreference losses in the multitask objective.","marker":"Cipolla et al. (2018)"}],"fun_headline_variants":["Soft targets lift coreference F1 past 67.2 baseline","Span-scoring with soft labels beats baseline resolver","Weighted loss improves mention net for coreference","Soft-target loss lifts coreference F1 beyond baseline","Span model with soft targets edges baseline coreference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that unannotated mentions in coreference data are roughly the same distribution as annotated ones, so the true mention distribution can be learned by extrapolating from labeled samples.","fun_headline_variants_meta":{"raw":{"variants":["Soft targets lift coreference F1 past 67.2 baseline","Span-scoring with soft labels beats baseline resolver","Weighted loss improves mention net for coreference","Soft-target loss lifts coreference F1 beyond baseline","Span model with soft targets edges baseline coreference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000462,"raw_usage":{"total_tokens":2226,"prompt_tokens":774,"completion_tokens":1452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":390,"completion_tokens_details":{"reasoning_tokens":1376}},"tokens_in":390,"tokens_out":1452,"duration_ms":9023,"temperature":1.0,"reasoning_tokens":1376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:08:51.748270+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fully annotate all mentions in a held-out portion of the OntoNotes test set, train the soft-target span-scoring model on the partially annotated data, and compare its detections against the full annotations. If the recall gain over the baseline consists mostly of spans that are true mentions, the distributional assumption holds; if the gain is mostly false positives or differs by mention type (e.g., pronouns vs. nominals), the assumption fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CoNLL 2012 OntoNotes dataset and the average F1 metric used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the uncertainty-based method for weighting the mention-detection and coreference losses in the multitask objective."}],"review_version":1}