CrossWeigh: Training Named Entity Tagger from Imperfect Annotations

Jiacheng Liu; Jiawei Han; Jingbo Shang; Lihao Lu; Liyuan Liu; Zihan Wang

REVIEW 3 major objections 6 minor 1 cited by

CrossWeigh: Training Named Entity Tagger from Imperfect Annotations

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that noisy gold labels in NER are common enough to hurt taggers, and that CrossWeigh's entity-disjoint cross-checking plus exponential downweighting converts those mistakes into consistent test-F1 gains.

desk verdict A durable corrected CoNLL03 test set plus a plausible weighted-training recipe; the mechanism story is softer than the numbers, but the benchmark alone is worth a serious referee. read the letter →

arxiv 1909.01441 v1 pith:EZW34JTV submitted 2019-09-03 cs.CL

Zihan Wang , Jingbo Shang , Liyuan Liu , Lihao Lu , Jiacheng Liu , Jiawei Han This is my paper · ORCID

classification cs.CL

keywords namedentityrecognitionlabelnoiseannotationerrorstrainingdatareweightingCoNLL03cross-validationsequencetaggingnoisylabels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Named entity recognition models are trained on hand-annotated data, and hand annotation is imperfect. This paper shows that in the standard CoNLL03 benchmark, about 5.38% of test sentences contain label mistakes, manually corrects those mistakes into a cleaner test set, and proposes CrossWeigh, a training-time procedure that lets the tagger itself flag likely annotation errors and lowers their weight. Plugging CrossWeigh into several base taggers improves test F1 on the original and corrected CoNLL03 sets, on an emerging-entity dataset, and on a low-resource language dataset, while also reducing run-to-run variance. The point is that annotation noise is a measurable, addressable obstacle in NER, not just a background irritation.

What carries the argument

The central mechanism is the mistake-estimation loop, a $k$-fold cross-check made honest by entity-disjoint filtering: before scoring a held-out fold, every training sentence containing any surface entity string that appears in that fold is removed, so the scorer has no way to memorize the entity from training. Repeated disagreement votes accumulate into a per-sentence confidence $c_i$, and the exponential weight $w_i=\epsilon^{c_i}$ converts those votes into a training weight for the final tagger. The reweighting step is what carries the performance gain; the entity-disjoint filtering is what keeps the votes from just reflecting how well the model memorized familiar entity strings.

What would settle it

Run CrossWeigh's estimator on a training set whose labels are known to be clean, or whose errors are injected at known positions; if the flagged sentences do not concentrate on the injected errors, or if downweighting flagged but verified-correct sentences still produces the same F1 gain, then the disagreement signal is measuring example difficulty rather than annotation noise, and the claimed mechanism is wrong.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that disagreement under entity-disjoint cross-checking is usable evidence of label noise. The training sentences are split into $k$ folds; for each fold, a model is trained on every sentence that shares no surface entity string with the held-out fold, then predicts that fold; sentences whose stored label disagrees with the prediction receive one vote. After $t$ random repartitions, sentence $x_i$ has confidence $c_i$ equal to its vote count, and the final model is trained with weight $w_i=\epsilon^{c_i}$, where $\epsilon$ is chosen from a rough estimate of detection precision. In the authors' experiments this consistently raises test F1: Flair goes from 92.87 to 93.19 on original CoNLL03 and from 93.89 to 94.18 on the corrected test set, and similar gains appear for other models and on WNUT'17 and Sinhalese data.

Load-bearing premise

The load-bearing assumption is that a model trained without seeing any of a sentence's entity strings disagrees with the stored label because the label is wrong, not because the sentence is simply hard; if hard-but-correct sentences dominate the disagreement votes, CrossWeigh would be downweighting exactly the examples the model needs to learn from.

Editorial extensions

If this is right

Training with CrossWeigh raises test F1 for every base NER model the paper tries, by roughly 0.2 to 0.4 points on CoNLL03, with equal or smaller standard deviations.
The corrected CoNLL03 test set yields higher F1 scores and smaller variances for all re-evaluated models, so model rankings may change when annotation noise is removed.
CrossWeigh transfers to settings it was not designed for: emerging-entity recognition (WNUT'17) and a low-resource language (Sinhalese), suggesting the mechanism is not specific to CoNLL03's error patterns.
The noise detector finds more than 75 percent of the true label mistakes in the test set, so the same machinery can be used to prioritize human review during annotation rather than only to reweight training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the disagreement signal is genuinely about label noise, CrossWeigh-style reweighting should transfer to other sequence-labeling tasks with noisy gold labels, such as part-of-speech tagging or medical entity extraction, where the entity-disjoint filter would become a span-disjoint or token-disjoint filter.
The low detection precision (about 25 percent) suggests the method's value is in concentrating reweighting rather than in deleting sentences; a natural test is to compare CrossWeigh with hard removal of all flagged sentences and with tuning the weight floor.
The paper's own future direction, an iterative or meta-learned version, could turn the one-shot reweighting into a boosting-like loop that progressively burns in corrections; that is a testable extension rather than a claim the paper makes.
Because models behave differently on the original and corrected test sets, a practical extension is to report NER results on both; the gap between the two is itself a cheap measure of how annotation-noise-sensitive a model is.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the corrected CoNLL03 test set is the most durable contribution here, and the re-evaluation of six models on it is careful, released, and likely to be reused. The CrossWeigh training framework is a reasonable engineering contribution with consistent small gains, but the paper's claim that those gains come from finding label mistakes is not fully supported; the data allow a hardness-reweighting story.

What is genuinely new: five annotators double-checked every test sentence, 186 sentences were corrected, agreement was 95.66%, and the corrected set is public. Re-evaluation shows higher F1 and lower variance across models, which makes sense if the original test set contained a few misleading labels. The entity-disjoint k-fold filtering in CrossWeigh is a specific new mechanism, and the ablation shows it matters: without it the gains vanish, and it beats random discard by a small margin. The gains also appear across three base models and two additional datasets, and the code is released. This is honest, reproducible empirical work.

Soft spots, in proportion: the word "significant" is used without significance tests, and the improvements are small enough that overlapping standard deviations make that word do too much work. Hyperparameters k, t, and epsilon were selected with the test set in view in Section 4.5; epsilon has a plausible manual justification, but k and t are chosen partly by test F1. The deeper issue is mechanistically important: Table 9 reports detection precision around 25%, so roughly three-quarters of flagged sentences are actually correct. If those false positives are biased toward hard-but-correct examples rather than uniform noise, then Eq. 4 with epsilon=0.7 is down-weighting informative training sentences, and the consistent F1 gains could be generic hardness suppression rather than evidence of detecting label noise. The ablation in Table 4 controls entity-disjoint filtering but does not control the weight distribution itself: there is no baseline that applies the same weights to randomly chosen or uncertainty-selected sentences. Given the small deltas, this is a material missing control, not a pedantic point. It is not an internal contradiction; it is an over-strong interpretation of the evidence.

The citation pattern is fine, and the distinction from earlier POS-tagging noise detection and from boosting is drawn accurately. Who is this for: NER practitioners, benchmark builders, and anyone comparing models on CoNLL03 — the corrected test set is a real resource. The paper deserves a serious referee and likely a revision that adds validation-only hyperparameter selection, significance testing, and a control for the reweighting signal itself. I would cite it for the corrected test set and would bring it to a reading group for a discussion of what the framework is actually doing.

Referee Report

3 major / 6 minor

Summary. The paper has two parts. First, it manually re-annotates the CoNLL03 test set, identifies 186 sentences (5.38%) with label mistakes, and releases a corrected test set; re-evaluating six NER models on this set yields higher and more stable F1 than on the original set. Second, it proposes CrossWeigh, a training-time framework that partitions the training data into k folds, trains fold models on entity-disjoint subsets, counts how many times each sentence’s label disagrees with the fold models (c_i), and then trains a final model with per-sentence weights ε^{c_i}. Experiments report F1 gains over VanillaNER, Flair, and Pooled-Flair on CoNLL03, and for Flair/BERT on WNUT'17 and Sinhalese, together with an analysis of CrossWeigh’s ability to flag actual mistakes.

Significance. The corrected CoNLL03 test set is a useful resource, and the re-evaluation with five runs per model is carefully done; the released code and corrected data support reproducibility. CrossWeigh is simple, model-agnostic, and shows consistent positive deltas in all reported settings, including two out-of-domain datasets with fixed hyperparameters. If the effect is truly due to detecting label noise, this would be a valuable contribution. The main weakness is that the experiments do not yet rule out a hardness/regularization confound: with detection precision around 25%, most down-weighted sentences are not annotation errors, and the method has not been compared against a control that re-weights random or uncertainty-selected sentences. The test-set-based choice of k, t, and ε also inflates the CoNLL03 numbers. These issues are addressable, so the contribution is promising but not yet firmly established.

major comments (3)

[Section 3.3 and Table 9] The load-bearing assumption is that disagreement of an entity-disjoint fold model with the stored label is a valid signal for label noise. Table 9 shows Flair’s detection precision is 25.1% (144/573 flagged test sentences are confirmed mistakes), so about three-quarters of the down-weighted sentences are correct. If those false positives are skewed toward hard-but-correct examples, the F1 gains in Tables 3 and 10 could come from down-weighting difficult examples rather than from noise removal. This is not merely pedantic: the reported in-domain gains are 0.15–0.34 F1, and the entity-disjoint filtering in Eq. (3) contributes directly by forcing fold models to predict surface forms never seen in training, which is exactly when hard examples are expected to cause disagreement. The current controls (Table 4) only vary the filtering step; there is no control that applies the same weight distribution to randomly chosen sentences or to uncertainty-selected sentences. I ask for such controls, plus a synthetic-noise experiment that varies the corruption rate and checks whether CrossWeigh’s benefit tracks true noise level rather than general hardness.
[Section 4.5 and Table 3] The default hyperparameters (k=10, t=3, ε=0.7) are selected with reference to test F1 on the CoNLL03 test set, and Table 3 reports gains on the same test set. In fact, ε=0.5 is at least on par with ε=0.7 on the original test set (93.21 vs. 93.19), and k=5 is very close to k=10 (93.12 vs. 93.19), so the reported configuration is not a uniquely supported choice. The main CoNLL03 result should be re-run with hyperparameters selected on development data or fixed a priori, and the corrected set should be used only for final evaluation; otherwise the headline improvement is partially a selection artifact. The fixed-parameter results on WNUT'17 and Sinhalese mitigate this concern for the general framework but do not remove it for the CoNLL03 claim.
[Section 4 and Table 10] The paper claims 'significant improvements' and reports means with standard deviations over five runs, but no significance tests are provided. For example, Table 10 gives WNUT'17 F1 of 48.96 ± 0.97 without CrossWeigh and 50.03 ± 0.40 with CrossWeigh; the 1.07-point difference is not obviously significant at the 5% level with n=5. Please report paired significance tests or bootstrap confidence intervals on the per-run differences, and consider additional runs or bootstrapping for the smaller gains in Table 3 as well.

minor comments (6)

[Section 2.1] Please specify who performed the final verification round and how disagreements were adjudicated (e.g., majority vote, discussion, or a third expert), and report per-type error counts (boundary errors vs. type errors).
[Table 11 and caption] 'CrossWeight' in the caption is a typo for 'CrossWeigh', and 'ORD' inside the table cells should presumably be 'ORG'.
[Section 4.4] The fractional counts (e.g., 821.67 potential mistakes for VanillaNER) come from averaging over three runs; please report raw counts or explain the averaging convention.
[Section 3.4] The sentence '1−p represents the fraction of these detected label mistakes that might be still useful' is ambiguous; it should say 'the fraction of flagged sentences that are not actually mistakes'.
[Algorithm 1 and Eq. (1)] Please state explicitly that each fold model Mi is trained with uniform weights while the final model is trained on the full weighted training set; this is clear from the algorithm but the surrounding text could make the distinction more direct.
[Table 2] Please state that 'Corrected' means evaluating on the corrected test set while training on the original train/dev set, to avoid ambiguity with the corrected dataset mentioned in Table 3.

Circularity Check

1 steps flagged · score 4.0 of 10

Minor fitted-input circularity: in-domain F1 gains are partly self-selected because k and t were chosen on the same CoNLL03 test sets later used as evidence; the central framework remains independently grounded.

fitted input called prediction [Section 4.1 (Default Parameters), Section 4.5 (Tables 6-8), and Section 4.2 (Table 3)]
"For CrossWeigh parameters, by default, we set k = 10, t = 3, and ϵ = 0.7. ... In our experiments (see Table 6), we find that t = 3 provides a good enough result. ... As shown in Table 7, we observe that k = 5, k = 10 are significantly better than k = 2. ... As shown in Table 3, compared with the three algorithms, applying CrossWeigh always leads to a higher F1 score."

The default k and t are selected by comparing F1 on the same CoNLL03 original/corrected test sets that are later used as the evidence for improvement: Tables 6-8 report results under 'Original' and 'Corrected' for each hyperparameter, and Table 3 reports F1 on those exact same sets as the demonstration that 'CrossWeigh can improve the performance of NER models.' Thus the headline in-domain numbers are optimistically selected values of the evaluation metric, not independent predictions: the configuration was chosen because it scored well on the very numbers used to validate it. This is a selection-bias form of fitted input, not a definitional identity, and the central claim retains independent support because the same defaults also improve WNUT'17 and Sinhalese.

full rationale

The core derivation chain is self-contained and not circular: c_i is obtained from entity-disjoint fold models (Eq. 2-3), weights are set by w_i = epsilon^c_i (Eq. 4), and the final model is trained on the weighted objective (Eq. 1). No step defines X in terms of Y, no self-citation supplies a load-bearing uniqueness claim, and no ansatz is smuggled in by citation. The manually corrected test set is an independent, human-produced artifact rather than an output of CrossWeigh. The only circularity-adjacent issue is experimental: k and t (and, to a lesser extent, epsilon) were tuned using F1 on the CoNLL03 test sets, so the exact in-domain gains in Table 3 are partly self-selected. This does not reduce the method to its inputs by construction, and the out-of-domain results (WNUT'17, Sinhalese) with the same defaults provide independent evidence for the central claim. The hardness-confound concern raised by the reader is a validity risk about whether disagreements indicate noise or difficulty, but it is not a circularity in the derivation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on three tuned hyperparameters (epsilon, k, t), on domain assumptions about the correctness of the manual test-set correction, and on the ad hoc design assumption that entity-disjoint models provide a valid noise signal. No new physical or theoretical entities are introduced.

free parameters (3)

epsilon = 0.7 (default)
Weight scaling for sentences flagged as potential mistakes; set to 1-p where p is the estimated precision of mistake detection from a 100-sentence manual sample (Section 3.4, Section 4.1). Other values (0.3, 0.5, 0.9) were tested and 0.5/0.7 performed best (Table 8).
k = 10 (default)
Number of folds in mistake estimation; chosen by parameter study on test F1 (Table 7): k=5 and k=10 outperform k=2.
t = 3 (default)
Number of estimation iterations; chosen by parameter study on test F1 (Table 6).

assumptions (4)

domain assumption The MUC-based CoNLL annotation guideline is the correct reference for judging whether original labels are mistakes; cases that deviate consistently from the guideline but are treated as correct (e.g., home/away team asymmetry) are left untouched.
The test-set correction in Section 2.1 relies on this standard to decide that 186 sentences contain errors and to leave ambiguous but consistent patterns uncorrected (Section 5).
domain assumption The human correction process, with two independent annotators per sentence and one verification pass at 95.66% agreement, yields an accurate ground-truth test set.
The re-evaluation and the reported improvement on the corrected set assume the corrected labels are correct; Section 2.1 describes this protocol.
ad hoc to paper In each fold, removing every sentence that contains any surface entity appearing in the held-out fold (Eq. 3) makes the trained model's prediction a valid signal for whether the held-out label is wrong, rather than merely a signal for entity difficulty.
This is the core design assumption of CrossWeigh (Section 3.3); if disagreements were dominated by entity hardness, the reweighting would discard useful data. Ablations in Table 4 show the step is critical, but do not prove the assumption itself.
domain assumption The base NER models can consume a weighted training set with a per-sentence weight in the loss function (Eq. 1).
The framework's generality claim requires this; all experiments use models that implement weighted negative log-likelihood.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CrossWeigh: Training Named Entity Tagger from Imperfect Annotations." pith.science (2026). https://pith.science/paper/EZW34JTV

@misc{pith2026190901441,
  author       = {Pith},
  title        = {Pith review of: CrossWeigh: Training Named Entity Tagger from Imperfect Annotations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EZW34JTV}},
  note         = {Machine review of arXiv:1909.01441}
}

read the original abstract

Everyone makes mistakes. So do human annotators when curating labels for named entity recognition (NER). Such label mistakes might hurt model training and interfere model comparison. In this study, we dive deep into one of the widely-adopted NER benchmark datasets, CoNLL03 NER. We are able to identify label mistakes in about 5.38% test sentences, which is a significant ratio considering that the state-of-the-art test F1 score is already around 93%. Therefore, we manually correct these label mistakes and form a cleaner test set. Our re-evaluation of popular models on this corrected test set leads to more accurate assessments, compared to those on the original test set. More importantly, we propose a simple yet effective framework, CrossWeigh, to handle label mistakes during NER model training. Specifically, it partitions the training data into several folds and train independent NER models to identify potential mistakes in each fold. Then it adjusts the weights of training data accordingly to train the final NER model. Extensive experiments demonstrate significant improvements of plugging various NER models into our proposed framework on three datasets. All implementations and corrected test set are available at our Github repo: https://github.com/ZihanWangKi/CrossWeigh.

Figures

Figures reproduced from arXiv: 1909.01441 by the authors.

**Figure 1.** An overview of our proposed CrossWeigh framework. It can better handle label mistakes, identify low quality annotations and conduct learning from a weighted training set. tifies the potential label mistakes in training data through a cross checking process and (2) mistake re-weighing: it lowers the weights of these instances during the training of the final NER model. The cross checking process is inspired by the k… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Tasks
cs.CL 2025-05 conditional novelty 5.0 of 10

A survey that categorizes multilingual prompting techniques by NLP task and language family, and designates potential state-of-the-art prompting methods for each dataset.

Reference graph

Works this paper leans on

23 extracted references · 20 canonical work pages · cited by 1 Pith paper

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Steven Abney, Robert E Schapire, and Yoram Singer. 1999. Boosting applied to tagging and pp attachment. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora

work page 1999
[4]

Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. In NAACL

work page 2019
[5]

Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. https://aclanthology.info/papers/C18-1139/c18-1139 Contextual string embeddings for sequence labeling . In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018 , pages 1638--1649

work page 2018
[6]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018
[7]

Yoav Freund, Robert Schapire, and Naoki Abe. 1999. A short introduction to boosting. Journal-Japanese Society For Artificial Intelligence, 14(771-780):1612

work page 1999
[8]

Sigr \' u n Helgad \' o ttir, Hrafn Loftsson, and Eir \' kur R \" o gnvaldsson. 2014. http://www.lrec-conf.org/proceedings/lrec2014/summaries/677.html Correcting errors in a new gold standard for tagging icelandic text . In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 201...

work page 2014

Show all 23 references

[9]

Ron Kohavi. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In IJCAI , pages 1137--1145. Morgan Kaufmann

1995
[10]

Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. http://aclweb.org/anthology/N/N16/N16-1030.pdf Neural architectures for named entity recognition . In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Asso...

2016
[11]

Liyuan Liu, Xiang Ren, Jingbo Shang, Xiaotao Gu, Jian Peng, and Jiawei Han. 2018 a . Efficient contextualized representation: Language model pruning for sequence labeling. In EMNLP , pages 1215--1225. Association for Computational Linguistics

2018
[12]

Liyuan Liu, Jingbo Shang, Xiang Ren, Frank Fangzheng Xu, Huan Gui, Jian Peng, and Jiawei Han. 2018 b . Empower sequence labeling with task-aware neural language model. In Thirty-Second AAAI Conference on Artificial Intelligence

2018
[13]

Xuezhe Ma and Eduard H. Hovy. 2016. http://aclweb.org/anthology/P/P16/P16-1101.pdf End-to-end sequence labeling via bi-directional lstm-cnns-crf . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, G...

2016
[14]

Christopher D. Manning. 2011. https://doi.org/10.1007/978-3-642-19400-9\_14 Part-of-speech tagging from 97 \ In Computational Linguistics and Intelligent Text Processing - 12th International Conference, CICLing 2011, Tokyo, Japan, February 20-26, 2011. Proceedings, Part I , pa...

2011 doi
[15]

Tetsuji Nakagawa and Yuji Matsumoto. 2002. Detecting errors in corpora using support vector machines. In Proceedings of the 19th international conference on Computational linguistics-Volume 1, pages 1--7. Association for Computational Linguistics

2002
[16]

Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power

Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In ACL (1) , pages 1756--1765. Association for Computational Linguistics

2017
[17]

Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT , pages 2227--2237. Association for Computational Linguistics

2018
[18]

J \"o rgen Pind, Fri rik Magn \'u sson, and Stef \'a n Briem. 1991. \'I slensk or t \' nib \'o k [the icelandic frequency dictionary]. The Institute of Lexicography, University of Iceland, Reykjavik, Iceland

1991
[19]

Ines Rehbein and Josef Ruppenhofer. 2017. https://doi.org/10.18653/v1/P17-1107 Detecting annotation noise in automatically labelled data . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August ...

2017 doi
[20]

Tjong Kim Sang and Fien De Meulder

Erik F. Tjong Kim Sang and Fien De Meulder. 2003. http://aclweb.org/anthology/W/W03/W03-0419.pdf Introduction to the conll-2003 shared task: Language-independent named entity recognition . In Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held ...

2003
[21]

Robert E Schapire and Yoram Singer. 1999. Improved boosting algorithms using confidence-rated predictions. Machine learning, 37(3):297--336

1999
[22]

Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity tagger using domain-specific dictionary. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2054--2064

2018
[23]

Amarnag Subramanya, Slav Petrov, and Fernando C. N. Pereira. 2010. Efficient graph-based semi-supervised learning of structured tagging models. In EMNLP , pages 167--176. ACL

2010

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Steven Abney, Robert E Schapire, and Yoram Singer. 1999. Boosting applied to tagging and pp attachment. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora

work page 1999

[4] [4]

Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. In NAACL

work page 2019

[5] [5]

Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. https://aclanthology.info/papers/C18-1139/c18-1139 Contextual string embeddings for sequence labeling . In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018 , pages 1638--1649

work page 2018

[6] [6]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018

[7] [7]

Yoav Freund, Robert Schapire, and Naoki Abe. 1999. A short introduction to boosting. Journal-Japanese Society For Artificial Intelligence, 14(771-780):1612

work page 1999

[8] [8]

Sigr \' u n Helgad \' o ttir, Hrafn Loftsson, and Eir \' kur R \" o gnvaldsson. 2014. http://www.lrec-conf.org/proceedings/lrec2014/summaries/677.html Correcting errors in a new gold standard for tagging icelandic text . In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 201...

work page 2014