{"id":"ab7149ac-1e1d-42e9-b480-b95f693812eb","arxiv_id":"1909.01441","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A corrected CoNLL03 NER test set and a cross-validation-based reweighting method, CrossWeigh, that improves NER accuracy by down-weighting likely mislabeled training sentences.","lead":"The authors found label mistakes in about 5.38% of sentences in the widely used CoNLL03 named entity recognition test set, manually corrected them, and released a cleaner benchmark. They also introduce CrossWeigh, a training framework that detects likely annotation errors and down-weights them, yielding small but consistent F1 gains across several NER models and datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CrossWeigh's reweighting signal may be a hardness confound: Table 9 shows 75% of flagged test sentences are correct, so F1 gains could come from generic down-weighting of hard examples rather than label-mistake detection.","rationale":"I read the paper as making two coupled claims: CrossWeigh detects annotation mistakes with reasonable recall and precision, and because of that detection, plugging it into NER models yields consistent F1 gains. The reader's conditional verdict already identifies the entity-disjoint filtering assumption as the weakest spot, and I agree. The reweighting rule w_i = epsilon^c_i only helps if c_i correlates with mistake likelihood; Section 4.4's own numbers show a large pool of false positives, so the mechanism could instead be lowering the influence of genuinely hard sentences after the entity-disjoint filtering has deliberately made them harder to predict. That could still improve test F1 in some settings, but it would change what CrossWeigh demonstrates and weaken the 'imperfect annotations' framing. I did not find a fatal flaw or an internal inconsistency: the ablations, cross-dataset replications, and released code are real supporting evidence, and the direction of the gains is consistent across eight reported settings. The missing piece is a control that removes label correctness from the reweighting decision. The proposed uncertainty-based hardness baseline provides that control cheaply with the existing implementation. If the control matches CrossWeigh's numbers, the verdict should remain conditional with a revised interpretation; if the control fails to match, the label-noise mechanism is supported. Either way the reader's conditional verdict is appropriate, so I recommend no change to the verdict.","tokens_in":12318,"tokens_out":7895,"duration_ms":89486,"concrete_test":"Using the released code, train Flair on CoNLL03 with the exact CrossWeigh pipeline, but in Section 3.3 replace the binary signal y_j != y_hat_j with a label-free hardness score, such as the negative log-likelihood of the fold model on the same held-out fold, threshold it to flag a comparable number of training sentences, and apply the same Eq. 4 reweighting schedule. If test F1 still reaches roughly 93.19 on the original set and 94.18 on the corrected set, the improvements are explained by generic down-weighting of hard examples and the label-mistake mechanism is not established; if F1 drops back toward the baselines of 92.87 and 93.89, the disagreement signal is carrying genuine label-noise information beyond hardness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3's estimator flags a sentence whenever a fold model trained on the entity-disjoint set (Eq. 3) disagrees with the stored label. Both the training-time benefit and the Section 4.4 identification claim require that such disagreement is enriched for true annotation errors. Table 9 gives Flair detection precision of 25.1% (144/573) on the corrected test set, meaning roughly three-quarters of flagged sentences are not mistakes. If those false positives are biased toward hard-but-correct examples rather than uniformly distributed, then Eq. 4 with epsilon=0.7 actively down-weights informative training sentences, and the consistent F1 gains in Tables 3 and 10 could be a generic regularization/hardness-suppression effect instead of evidence that CrossWeigh finds label noise. Table 4 only ablates the entity-disjoint filtering step, with the Random Discard control, and does not control the reweighting signal itself; no baseline applies the same weight distribution to randomly chosen or uncertainty-selected sentences. Because the reported gains are small, ranging from 0.15 to 1.34 F1 points, a hardness confound is material rather than pedantic. This is not an internal inconsistency, but it is the least secure premise connecting the framework's design to the stated claim of handling imperfect annotations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper has two parts. First, it manually re-annotates the CoNLL03 test set, identifies 186 sentences (5.38%) with label mistakes, and releases a corrected test set; re-evaluating six NER models on this set yields higher and more stable F1 than on the original set. Second, it proposes CrossWeigh, a training-time framework that partitions the training data into k folds, trains fold models on entity-disjoint subsets, counts how many times each sentence’s label disagrees with the fold models (c_i), and then trains a final model with per-sentence weights ε^{c_i}. Experiments report F1 gains over VanillaNER, Flair, and Pooled-Flair on CoNLL03, and for Flair/BERT on WNUT'17 and Sinhalese, together with an analysis of CrossWeigh’s ability to flag actual mistakes.","tokens_in":12636,"tokens_out":9627,"duration_ms":90469,"significance":"The corrected CoNLL03 test set is a useful resource, and the re-evaluation with five runs per model is carefully done; the released code and corrected data support reproducibility. CrossWeigh is simple, model-agnostic, and shows consistent positive deltas in all reported settings, including two out-of-domain datasets with fixed hyperparameters. If the effect is truly due to detecting label noise, this would be a valuable contribution. The main weakness is that the experiments do not yet rule out a hardness/regularization confound: with detection precision around 25%, most down-weighted sentences are not annotation errors, and the method has not been compared against a control that re-weights random or uncertainty-selected sentences. The test-set-based choice of k, t, and ε also inflates the CoNLL03 numbers. These issues are addressable, so the contribution is promising but not yet firmly established.","major_comments":[{"comment":"The load-bearing assumption is that disagreement of an entity-disjoint fold model with the stored label is a valid signal for label noise. Table 9 shows Flair’s detection precision is 25.1% (144/573 flagged test sentences are confirmed mistakes), so about three-quarters of the down-weighted sentences are correct. If those false positives are skewed toward hard-but-correct examples, the F1 gains in Tables 3 and 10 could come from down-weighting difficult examples rather than from noise removal. This is not merely pedantic: the reported in-domain gains are 0.15–0.34 F1, and the entity-disjoint filtering in Eq. (3) contributes directly by forcing fold models to predict surface forms never seen in training, which is exactly when hard examples are expected to cause disagreement. The current controls (Table 4) only vary the filtering step; there is no control that applies the same weight distribution to randomly chosen sentences or to uncertainty-selected sentences. I ask for such controls, plus a synthetic-noise experiment that varies the corruption rate and checks whether CrossWeigh’s benefit tracks true noise level rather than general hardness.","section":"Section 3.3 and Table 9"},{"comment":"The default hyperparameters (k=10, t=3, ε=0.7) are selected with reference to test F1 on the CoNLL03 test set, and Table 3 reports gains on the same test set. In fact, ε=0.5 is at least on par with ε=0.7 on the original test set (93.21 vs. 93.19), and k=5 is very close to k=10 (93.12 vs. 93.19), so the reported configuration is not a uniquely supported choice. The main CoNLL03 result should be re-run with hyperparameters selected on development data or fixed a priori, and the corrected set should be used only for final evaluation; otherwise the headline improvement is partially a selection artifact. The fixed-parameter results on WNUT'17 and Sinhalese mitigate this concern for the general framework but do not remove it for the CoNLL03 claim.","section":"Section 4.5 and Table 3"},{"comment":"The paper claims 'significant improvements' and reports means with standard deviations over five runs, but no significance tests are provided. For example, Table 10 gives WNUT'17 F1 of 48.96 ± 0.97 without CrossWeigh and 50.03 ± 0.40 with CrossWeigh; the 1.07-point difference is not obviously significant at the 5% level with n=5. Please report paired significance tests or bootstrap confidence intervals on the per-run differences, and consider additional runs or bootstrapping for the smaller gains in Table 3 as well.","section":"Section 4 and Table 10"}],"minor_comments":[{"comment":"Please specify who performed the final verification round and how disagreements were adjudicated (e.g., majority vote, discussion, or a third expert), and report per-type error counts (boundary errors vs. type errors).","section":"Section 2.1"},{"comment":"'CrossWeight' in the caption is a typo for 'CrossWeigh', and 'ORD' inside the table cells should presumably be 'ORG'.","section":"Table 11 and caption"},{"comment":"The fractional counts (e.g., 821.67 potential mistakes for VanillaNER) come from averaging over three runs; please report raw counts or explain the averaging convention.","section":"Section 4.4"},{"comment":"The sentence '1−p represents the fraction of these detected label mistakes that might be still useful' is ambiguous; it should say 'the fraction of flagged sentences that are not actually mistakes'.","section":"Section 3.4"},{"comment":"Please state explicitly that each fold model Mi is trained with uniform weights while the final model is trained on the full weighted training set; this is clear from the algorithm but the surrounding text could make the distinction more direct.","section":"Algorithm 1 and Eq. (1)"},{"comment":"Please state that 'Corrected' means evaluating on the corrected test set while training on the original train/dev set, to avoid ambiguity with the corrected dataset mentioned in Table 3.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The corrected test set alone is a worthwhile artifact, and the paper is likely publishable after revision. The main risk is that CrossWeigh's improvements are a regularization effect rather than noise-specific learning; if the additional controls fail to distinguish CrossWeigh from random or uncertainty reweighting, the authors will need to substantially narrow the central claim. There is no concern about novelty disclosure: the released code and data match the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the corrected CoNLL03 test set is the most durable contribution here, and the re-evaluation of six models on it is careful, released, and likely to be reused. The CrossWeigh training framework is a reasonable engineering contribution with consistent small gains, but the paper's claim that those gains come from finding label mistakes is not fully supported; the data allow a hardness-reweighting story.\n\nWhat is genuinely new: five annotators double-checked every test sentence, 186 sentences were corrected, agreement was 95.66%, and the corrected set is public. Re-evaluation shows higher F1 and lower variance across models, which makes sense if the original test set contained a few misleading labels. The entity-disjoint k-fold filtering in CrossWeigh is a specific new mechanism, and the ablation shows it matters: without it the gains vanish, and it beats random discard by a small margin. The gains also appear across three base models and two additional datasets, and the code is released. This is honest, reproducible empirical work.\n\nSoft spots, in proportion: the word \"significant\" is used without significance tests, and the improvements are small enough that overlapping standard deviations make that word do too much work. Hyperparameters k, t, and epsilon were selected with the test set in view in Section 4.5; epsilon has a plausible manual justification, but k and t are chosen partly by test F1. The deeper issue is mechanistically important: Table 9 reports detection precision around 25%, so roughly three-quarters of flagged sentences are actually correct. If those false positives are biased toward hard-but-correct examples rather than uniform noise, then Eq. 4 with epsilon=0.7 is down-weighting informative training sentences, and the consistent F1 gains could be generic hardness suppression rather than evidence of detecting label noise. The ablation in Table 4 controls entity-disjoint filtering but does not control the weight distribution itself: there is no baseline that applies the same weights to randomly chosen or uncertainty-selected sentences. Given the small deltas, this is a material missing control, not a pedantic point. It is not an internal contradiction; it is an over-strong interpretation of the evidence.\n\nThe citation pattern is fine, and the distinction from earlier POS-tagging noise detection and from boosting is drawn accurately. Who is this for: NER practitioners, benchmark builders, and anyone comparing models on CoNLL03 — the corrected test set is a real resource. The paper deserves a serious referee and likely a revision that adds validation-only hyperparameter selection, significance testing, and a control for the reweighting signal itself. I would cite it for the corrected test set and would bring it to a reading group for a discussion of what the framework is actually doing.","headline":"A durable corrected CoNLL03 test set plus a plausible weighted-training recipe; the mechanism story is softer than the numbers, but the benchmark alone is worth a serious referee.","tokens_in":13122,"tokens_out":1767,"would_cite":true,"duration_ms":19674,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that noisy gold labels in NER are common enough to hurt taggers, and that CrossWeigh's entity-disjoint cross-checking plus exponential downweighting converts those mistakes into consistent test-F1 gains.","keywords":["named entity recognition","label noise","annotation errors","training data reweighting","CoNLL03","cross-validation","sequence tagging","noisy labels"],"falsifier":"Run CrossWeigh's estimator on a training set whose labels are known to be clean, or whose errors are injected at known positions; if the flagged sentences do not concentrate on the injected errors, or if downweighting flagged but verified-correct sentences still produces the same F1 gain, then the disagreement signal is measuring example difficulty rather than annotation noise, and the claimed mechanism is wrong.","tokens_in":12124,"feed_emoji":"🏷️","tokens_out":8469,"duration_ms":74647,"temperature":0.7,"pith_summary":"Named entity recognition models are trained on hand-annotated data, and hand annotation is imperfect. This paper shows that in the standard CoNLL03 benchmark, about 5.38% of test sentences contain label mistakes, manually corrects those mistakes into a cleaner test set, and proposes CrossWeigh, a training-time procedure that lets the tagger itself flag likely annotation errors and lowers their weight. Plugging CrossWeigh into several base taggers improves test F1 on the original and corrected CoNLL03 sets, on an emerging-entity dataset, and on a low-resource language dataset, while also reducing run-to-run variance. The point is that annotation noise is a measurable, addressable obstacle in NER, not just a background irritation.","feed_headline":"Downweighting noisy labels lifts NER F1 scores","feed_subtitle":"A k-fold cross-check flags likely annotation mistakes and cuts their training weight on three datasets.","key_machinery":"The central mechanism is the mistake-estimation loop, a $k$-fold cross-check made honest by entity-disjoint filtering: before scoring a held-out fold, every training sentence containing any surface entity string that appears in that fold is removed, so the scorer has no way to memorize the entity from training. Repeated disagreement votes accumulate into a per-sentence confidence $c_i$, and the exponential weight $w_i=\\epsilon^{c_i}$ converts those votes into a training weight for the final tagger. The reweighting step is what carries the performance gain; the entity-disjoint filtering is what keeps the votes from just reflecting how well the model memorized familiar entity strings.","core_discovery":"On the paper's own terms, the discovery is that disagreement under entity-disjoint cross-checking is usable evidence of label noise. The training sentences are split into $k$ folds; for each fold, a model is trained on every sentence that shares no surface entity string with the held-out fold, then predicts that fold; sentences whose stored label disagrees with the prediction receive one vote. After $t$ random repartitions, sentence $x_i$ has confidence $c_i$ equal to its vote count, and the final model is trained with weight $w_i=\\epsilon^{c_i}$, where $\\epsilon$ is chosen from a rough estimate of detection precision. In the authors' experiments this consistently raises test F1: Flair goes from 92.87 to 93.19 on original CoNLL03 and from 93.89 to 94.18 on the corrected test set, and similar gains appear for other models and on WNUT'17 and Sinhalese data.","pith_inferences":["If the disagreement signal is genuinely about label noise, CrossWeigh-style reweighting should transfer to other sequence-labeling tasks with noisy gold labels, such as part-of-speech tagging or medical entity extraction, where the entity-disjoint filter would become a span-disjoint or token-disjoint filter.","The low detection precision (about 25 percent) suggests the method's value is in concentrating reweighting rather than in deleting sentences; a natural test is to compare CrossWeigh with hard removal of all flagged sentences and with tuning the weight floor.","The paper's own future direction, an iterative or meta-learned version, could turn the one-shot reweighting into a boosting-like loop that progressively burns in corrections; that is a testable extension rather than a claim the paper makes.","Because models behave differently on the original and corrected test sets, a practical extension is to report NER results on both; the gap between the two is itself a cheap measure of how annotation-noise-sensitive a model is."],"forward_implications":["Training with CrossWeigh raises test F1 for every base NER model the paper tries, by roughly 0.2 to 0.4 points on CoNLL03, with equal or smaller standard deviations.","The corrected CoNLL03 test set yields higher F1 scores and smaller variances for all re-evaluated models, so model rankings may change when annotation noise is removed.","CrossWeigh transfers to settings it was not designed for: emerging-entity recognition (WNUT'17) and a low-resource language (Sinhalese), suggesting the mechanism is not specific to CoNLL03's error patterns.","The noise detector finds more than 75 percent of the true label mistakes in the test set, so the same machinery can be used to prioritize human review during annotation rather than only to reweight training data."],"supporting_citations":[{"why":"It supplies the CoNLL03 NER dataset, its annotation guideline, and the train/dev/test split whose test labels the paper corrects and whose training labels CrossWeigh reweights.","marker":"Sang and Meulder, 2003"},{"why":"It supplies Flair, the contextual string-embedding tagger used as the main base model in the experiments.","marker":"Akbik et al., 2018"},{"why":"It supplies Pooled-Flair, the stronger baseline used to show that CrossWeigh gains are not tied to one tagging architecture.","marker":"Akbik et al., 2019"},{"why":"It supplies ELMo and the train-plus-development evaluation setup used in the experiments; ELMo is one of the models re-evaluated on the corrected test set.","marker":"Peters et al., 2018"},{"why":"It supplies the LSTM-CRF architecture used as a base NER model and as one of the re-evaluated algorithms.","marker":"Lample et al., 2016"},{"why":"It supplies the LSTM-CNNs-CRF architecture used as another base NER model and re-evaluated algorithm.","marker":"Ma and Hovy, 2016"},{"why":"It supplies BERT as the base encoder for the Sinhalese low-resource experiment, supporting the claim that CrossWeigh generalizes across languages.","marker":"Devlin et al., 2018"}],"fun_headline_variants":["CrossWeigh: vote on label noise, then downweight it","Disagreement reveals noisy labels; weigh them less","Cross-checked label weights boost NER F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a model trained without seeing any of a sentence's entity strings disagrees with the stored label because the label is wrong, not because the sentence is simply hard; if hard-but-correct sentences dominate the disagreement votes, CrossWeigh would be downweighting exactly the examples the model needs to learn from.","fun_headline_variants_meta":{"raw":{"variants":["CrossWeigh: vote on label noise, then downweight it","Disagreement reveals noisy labels; weigh them less","Cross-checked label weights boost NER F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1768,"prompt_tokens":961,"completion_tokens":807,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":754}},"tokens_in":577,"tokens_out":807,"duration_ms":8613,"temperature":1.0,"reasoning_tokens":754,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:17:33.656283+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CrossWeigh's estimator on a training set whose labels are known to be clean, or whose errors are injected at known positions; if the flagged sentences do not concentrate on the injected errors, or if downweighting flagged but verified-correct sentences still produces the same F1 gain, then the disagreement signal is measuring example difficulty rather than annotation noise, and the claimed mechanism is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies Flair, the contextual string-embedding tagger used as the main base model in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies Pooled-Flair, the stronger baseline used to show that CrossWeigh gains are not tied to one tagging architecture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the LSTM-CRF architecture used as a base NER model and as one of the re-evaluated algorithms."}],"review_version":1}