{"id":"997b5039-ffd8-4889-93ce-b97a59e25d3c","arxiv_id":"1908.08983","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":6,"one_line_summary":"An entity-targeted active learning strategy, combined with cross-lingual transfer and partial-CRF training, outperforms sequence-level and random baselines for low-resource NER.","lead":"This paper investigates the most efficient way to build a named entity recognition system for a language with very little labeled data. The authors find that starting from an English-trained model and then having humans label only the most uncertain name-like spans beats full-sentence annotation, reaching competitive accuracy with far fewer labeled tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'one-tenth of tokens' headline claim is unsupported: experiments annotate at most ~8% of LDC training tokens and ~2% of CoNLL tokens, and the 9.9 F1 figure is not reproducible from the reported tables.","rationale":"After reading the paper in full, the qualitative finding — ETAL with partial-CRF and cross-lingual transfer beats the active-learning baselines on all six datasets — appears robust: the bootstrap tests in Table 1 cover three budgets, the ablation of partial-CRF vs full-CRF is sensible, and the human experiments, though noisy, show efficiency gains. The reader's chosen weakest assumption (selection bias in partial-CRF) is not, in my view, the decisive risk: active learning is deliberately non-random, the partial-CRF likelihood marginalizes over unlabeled tokens in the standard way, and the held-out test set is an honest measure of generalization. The more load-bearing problem is the paper's central quantitative headline. The 'one-tenth of tokens' and the '9.9 F1 average improvement' are never tied to a specific, reported budget. The experiments stop at 4000 tokens (2400 for Spanish-LDC), which is 1.5-8.2% of the training sets listed in Table 3, never 10%. The 9.9 number cannot be recovered from Tables 11-16 at any common budget: at the final runs the average advantage to the best baseline is about 5.2 and to SAL about 11. The '5.2 behind SUPERVISED ALL' is also unverifiable because the supervised-all curves are only in figures. These are the numbers the abstract and contributions lead with; if they are not precisely defined, the paper's core message overstates what the experiments show. This is addressable with a clear budget definition and a re-analysis, so conditional acceptance remains appropriate.","tokens_in":18598,"tokens_out":13028,"duration_ms":117351,"concrete_test":"For each of the six datasets, locate the active-learning run where the cumulative annotated token count equals 10% of the training-set token count in Table 3; if no run reaches 10%, interpolate the F1 curves from Tables 11-16 or explicitly mark the claim unevaluable. At that point, compute (i) the average over datasets of ETAL's F1 advantage over the best active-learning baseline and (ii) the average gap to SUPERVISED ALL. If the 9.9 and 5.2 figures are not reproduced, or cannot be computed because the budget is unreached, the headline claim should be revised to reflect the actual annotation budgets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim ('with just one-tenth of tokens annotated... average improvement of 9.9 F1' and 'only 5.2 F1 behind SUPERVISED ALL') is not tied to the experimental protocol. Section 3.1 fixes the annotation budget at 200 tokens per run; the maximum runs (Tables 11-16) reach 4000 tokens for most datasets and 2400 for Spanish-LDC. Against the training-set token counts in Table 3 (Hindi 48604, Indonesian 55270, Spanish-LDC 31799, German 206846, Dutch 200059, Spanish-CoNLL 264715), 4000 tokens is 8.2% (Hindi), 7.2% (Indonesian), 1.9% (German), 2.0% (Dutch), 1.5% (Spanish-CoNLL), and 2400 is 7.5% (Spanish-LDC). None of the experiments annotates one-tenth of the training data. Moreover, the '9.9 F1 average improvement' is not recoverable from the reported tables: at the final runs, the average gap to the best per-dataset baseline is about 5.2 F1, and the average gap to SAL is about 11 F1; neither is 9.9. The '5.2 F1 behind SUPERVISED ALL' is similarly hard to verify because the supervised-all numbers are only shown in figures, not tables. Thus the headline quantitative takeaway, which is the main evidence that 'a little annotation does a lot of good,' is either defined against an unspecified baseline/budget or is an overstatement. This does not invalidate the qualitative conclusion that ETAL outperforms the baselines, but it makes the central claim's most visible numbers unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a bootstrapping recipe for low-resource NER: start from a cross-lingual transferred model, then iteratively select uncertain entity spans (ETAL) for annotation, and fine-tune with a partial-CRF objective. The authors report simulated active-learning experiments on six languages (Hindi, Indonesian, Spanish-LDC, Spanish-CoNLL, German, Dutch) and human annotation experiments for Hindi, Indonesian, and Spanish, comparing ETAL against sequence-level (SAL), span-level confidence (CFEAL), and random (RAND) baselines. The central quantitative claims are that with one-tenth of tokens annotated ETAL achieves on average 9.9 F1 improvement over baselines and is only 5.2 F1 behind a model trained on all labeled data.","tokens_in":19021,"tokens_out":8833,"duration_ms":77662,"significance":"If the quantitative claims were fully supported, this would be a useful contribution: it combines transfer learning and active learning in a practical recipe, evaluates on multiple languages including low-resource ones, includes real human annotation, and provides code and paired bootstrap significance tests. The breadth of datasets and the human-annotation experiments are particular strengths. The main caveat is that the headline numbers ('one-tenth', 9.9, 5.2) are not supported by the experimental protocol as reported, and the partial-CRF training ignores the outcome-dependent sampling introduced by ETAL. The qualitative conclusion that ETAL is competitive with fewer annotations is credible from the tables, but the paper's most visible claims need correction.","major_comments":[{"comment":"The abstract and Contribution 1 state that the method works 'with just one-tenth of training data' / 'one-tenth of tokens annotated,' but the experimental protocol never reaches this budget. Section 3.1 sets the budget at 200 tokens per active-learning run, and the final runs in Tables 11–16 annotate at most 4000 tokens (2400 for Spanish-LDC). Relative to the training-set sizes in Table 3, 4000 tokens is 8.2% of Hindi, 7.2% of Indonesian, 1.9% of German, 2.0% of Dutch, and 1.5% of Spanish-CoNLL; 2400 tokens is 7.5% of Spanish-LDC. None of the six datasets reaches 10% of training tokens, so the 'one-tenth' claim is not demonstrated. Please either extend the experiments to a 10% budget or rephrase the claim to state the actual fractions actually annotated.","section":"Abstract; §3.1; Table 3"},{"comment":"The claimed average improvement of 9.9 F1 is not recoverable from the reported tables. At the final run, the gap between ETAL+PARTIAL-CRF+CT and the best per-dataset baseline is about 2.2 (Hindi), 7.0 (Indonesian), 6.9 (Spanish-LDC), 4.1 (Spanish-CoNLL), 3.6 (German), and 7.5 (Dutch), averaging about 5.2 F1; no comparison in the tables yields 9.9. The per-dataset margins reported in the text (Hindi +9.3, Indonesian +9.0, Spanish-LDC +7.5, etc.) also do not match the final-run gaps; for example, the Hindi final-run gap to the best baseline is 2.2, not 9.3. The authors should specify exactly which run, which baseline, and which averaging procedure produce 9.9, or correct the number.","section":"§3.2.1; Tables 11–16"},{"comment":"The claim that the full recipe is on average only 5.2 F1 behind SUPERVISED ALL cannot be verified from the manuscript, because the SUPERVISED ALL scores appear only as horizontal lines in Figure 2 and are not tabulated anywhere. Please report the supervised-all F1 for each dataset and state the budget (for example, final run or one-tenth of tokens) at which the 5.2 gap is computed; if the comparison is at the final run, note that the final budget is not one-tenth of the training data.","section":"§3.2.1; Figure 2"},{"comment":"The partial-CRF likelihood marginalizes over unannotated tokens, which corresponds to treating them as missing at random. ETAL, however, selects spans according to the current model's uncertainty, so the annotated tokens are not a random subset: the missingness depends on model predictions and on the entity structure of the sentence. This outcome-dependent sampling is not discussed; it means the partial likelihood is misspecified for the data-generating process, and part of the gain over RAND could reflect adaptation to the selected span distribution. Because the final evaluation is on held-out test data, this does not invalidate the empirical ranking, but the authors should state the assumption and ideally add a diagnostic that matches the number or type of annotated spans across strategies to separate selection effects from the training objective.","section":"§2.3.2"}],"minor_comments":[{"comment":"Appendix A.4 refers to 'Figure 4' for the confidence intervals, but the results are in Table 4; please fix the cross-reference.","section":"§A.4"},{"comment":"The list of entity-count tables reads 'Tables 5, 6, 7, 7, 9, 10'; the duplicate 7 should be 8.","section":"§A.5"},{"comment":"Algorithm 1 contains dangling pseudo-code fragments (for example, 'for i←1...len(x),j = 1 do'); please clean up the formatting so that the loop structure is clear.","section":"Algorithm 1"},{"comment":"The abbreviation CFEAL is never expanded; please define it at first use (for example, confidence-field-estimation active learning).","section":"§3.2"},{"comment":"There are typos in the figure and table captions: 'ET AL' and 'PARTI AL-CRF' should be 'ETAL' and 'PARTIAL-CRF'.","section":"Figure 2 and Table 1"},{"comment":"The human annotation study uses only two annotators per language, and the Indonesian results are acknowledged to be affected by gold-label inconsistencies; the claim that annotators are more accurate under ETAL should be presented as preliminary evidence rather than a general conclusion.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study, but the abstract and contributions overstate the experiments. The 'one-tenth', 9.9, and 5.2 numbers are exactly the claims readers will quote, so they must be made accurate or explicitly defined. I would not reject; the tables suggest the qualitative message is likely correct. Please also ask the authors to provide the underlying supervised-all numbers or per-run tables so that the headline figures are auditable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a legitimate contribution to low-resource NER. The genuinely new piece is ETAL, a span-level active learning criterion that computes entropy over likely entity spans and aggregates it across the corpus, selecting uncertain, frequent spans. The authors combine this with cross-lingual transfer and partial-CRF training, and evaluate the full recipe systematically against sequence-level (SAL), confidence-field (CFEAL), and random baselines on six datasets, using both simulated and human annotation. The main result is credible: ETAL beats the baselines by 5–7 F1 at the end of the runs, and paired bootstrap resampling shows significance at 600 and 1200 tokens. The code is public. That is a solid empirical study, and the ablation studies clarify the contribution of each component.\n\nWhat is not clean is the headline quantification. The abstract and contributions say “with just one-tenth of tokens annotated” and claim an average improvement of 9.9 F1. The experiments fix a 200-token budget per run and run up to 4000 tokens (2400 for Spanish-LDC). For the CoNLL datasets that is about 2% of the training data; for the LDC sets it is 7–8%. The experiments never reach one-tenth of the training data. The 9.9 F1 number is not recoverable from the reported tables: at the final runs, the gap to the best per-dataset baseline averages about 5.2 F1, and the gap to SAL is around 11 F1. The “only 5.2 F1 behind SUPERVISED ALL” claim also depends on figures rather than tabulated values. So the most visible quantitative takeaway is not tied to the experimental protocol. That is an overstatement, and it should be corrected.\n\nA smaller issue: partial-CRF marginalizes over unannotated tokens, which implicitly treats them as missing at random. ETAL selects spans based on model uncertainty, so the training distribution is deliberately biased. The paper does not discuss this. It is a known hazard of active learning, and it does not invalidate the relative comparisons since all strategies use the same training objective, but a sentence in the limitations section would be appropriate.\n\nThe human annotation experiment uses only two annotators per language, which makes the strong claims about annotator accuracy fragile. That is a minor concern given the simulation results.\n\nThis paper deserves serious peer review. The method is clearly described, the comparison is broad, and the qualitative conclusion—entity-targeted annotation with partial-CRF is a good recipe—holds up. A reviewer should push for precise numbers in the abstract and a brief discussion of selection bias, but the core contribution is sound.","headline":"A solid empirical recipe for low-resource NER with a genuinely new span-level active learning strategy; the headline budget numbers are imprecise, but the qualitative result holds up.","tokens_in":19519,"tokens_out":6067,"would_cite":true,"duration_ms":47383,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross-lingual transfer plus targeted span annotation reaches near-supervised NER with one-tenth of the labels.","keywords":["low-resource named entity recognition","cross-lingual transfer learning","active learning","entity-targeted annotation","partial conditional random fields","annotation efficiency","bilingual word embeddings","bootstrapping"],"falsifier":"Train ETAL's spans but weight the partial-CRF likelihood with the inverse probability of span selection, or compare ETAL against a random span sampler that is forced to select the same number and length of entities; if the F1 advantage shrinks or disappears, the reported 9.9 F1 gain is an artifact of selection bias rather than of targeting informative entities.","tokens_in":1717,"feed_emoji":"🎯","tokens_out":2212,"duration_ms":81297,"temperature":0.7,"pith_summary":"The paper asks how to efficiently build a named entity recognizer for a language with very little labeled data. It argues that the best approach is to start with a model transferred from English and then spend the human annotation budget on only the short entity spans the model is most unsure about, rather than whole sentences. With just one-tenth of the tokens annotated, this entity-targeted active learning strategy outperforms sequence-level and span-level active learning baselines by an average of 9.9 F1, and the full recipe lands only 5.2 F1 behind a model trained on all labeled data. The authors also show through simulated and real human annotation that annotators make fewer errors and read less when asked to label a highlighted entity span than when asked to label an entire sequence.","feed_headline":"A tenth of the labels builds a near-full entity tagger","feed_subtitle":"Cross-lingual transfer plus targeted span annotation reaches within 5.2 F1 of fully supervised models.","key_machinery":"The load-bearing mechanism is ETAL, an entity-targeted active-learning scorer combined with partial-CRF training. ETAL computes, with the forward-backward algorithm, the marginal probability that each token begins or continues an entity, multiplies these into a span-level entity probability, and ranks candidate spans by the entropy of that probability, aggregated across the unlabeled corpus; a dynamic program keeps the span search from becoming exponential in sequence length. The resulting partially labeled sequences are trained with a constrained conditional random field that sums the likelihood over all labelings consistent with the annotated spans. A cross-lingually transferred model provides both the initial parameters and the uncertainty signal that drives selection.","core_discovery":"The central claim is that cross-lingual transfer and targeted span-level active learning are complementary, and that combining them with a partial-CRF training objective yields near-supervised NER accuracy at a fraction of the annotation cost. Starting from an English-trained model projected to the target language through bilingual embeddings, the proposed ETAL strategy selects, across the unlabeled corpus, the entity spans whose marginal probability of being an entity is most uncertain, aggregates that uncertainty by span type, and asks humans only to label those spans. Training then uses a constrained CRF that marginalizes over unannotated tokens instead of guessing their labels. In simulated experiments ETAL beats random, confidence-field, and sequence-level baselines on all six test datasets; with one-tenth of tokens annotated the full recipe is on average 5.2 F1 below a fully supervised upper bound, and in human annotation it reaches within 2.1 F1 of full-sequence annotation while using six times fewer annotated tokens.","pith_inferences":["If the recipe generalizes, the same pattern—use transfer to get a rough model, then spend human effort only where the model is uncertain—should apply to other sparse-span sequence labeling tasks such as event extraction or grammatical error detection, not just NER; testing that would require adapting the span entropy scorer to those label sets.","The paper's own numbers imply that the value of cross-lingual transfer is mostly in the first few hundred tokens; beyond roughly 1000 tokens, entity-targeted annotation alone catches up, so practitioners with larger budgets might skip transfer and annotate directly.","Because ETAL selects spans by model uncertainty, the partial-CRF likelihood treats unannotated tokens as missing at random, an assumption the selection rule itself violates; a debiased estimator or an experiment matching random spans for entity density would reveal how much of the 9.9 F1 gain is due to span targeting versus distribution shift.","A practical by-product is that the same pipeline could build entity gazetteers quickly, since ETAL surfaces roughly twice as many candidate entities per annotated token as full-sequence annotation in the human study."],"forward_implications":["With one-tenth of tokens annotated, ETAL plus partial-CRF plus cross-lingual transfer beats RAND, CFEAL, and SAL baselines by an average of 9.9 F1 across the test datasets, and is only 5.2 F1 behind a model trained on all labeled data.","Cross-lingual transfer alone contributes an average of 8.6 F1 over no transfer at the one-tenth budget, and the untransferred system closes the gap after roughly 1000 tokens, so transfer matters most at the very start of bootstrapping.","Using a full CRF that fills unannotated tokens with model predictions hurts recall substantially (by 11.0 points for Hindi), so the partial-CRF objective is necessary to realize ETAL's gains.","Human annotators label entity spans with higher oracle agreement than when annotating full sequences, and ETAL yields competitive test F1 with about six times fewer annotated tokens, with an average gap of 2.1 F1 against full-sequence annotation.","Fine-tuning the transferred model on newly acquired data is generally faster and better than retraining on the concatenated corpus, though for a language close to English, corpus augmentation plus fine-tuning helps most."],"supporting_citations":[{"why":"Supplies the cross-lingual transfer model and resources that initialize the NER model from English training data.","marker":"Xie et al. (2018)"},{"why":"Provides the BiLSTM-CNN-CRF architecture used for all NER models in the paper.","marker":"Ma and Hovy (2016)"},{"why":"Defines the confidence-field-estimation baseline (CFEAL) that selects least confident spans.","marker":"Culotta and McCallum (2004)"},{"why":"Defines the sequence-level active learning baseline (SAL) that selects least confident full sentences.","marker":"Culotta and McCallum (2005)"},{"why":"Establishes partial and constrained CRF training on incompletely annotated data, used as PARTIAL-CRF.","marker":"Tsuboi et al. (2008)"},{"why":"Motivates the constrained CRF decoder that marginalizes over unannotated tokens.","marker":"Bellare and McCallum (2007)"},{"why":"Provides the CSLS metric used to build the word-to-word translation dictionary for transfer.","marker":"Conneau et al. (2017b)"},{"why":"Provides the paired bootstrap resampling method used to test statistical significance.","marker":"Koehn (2004)"}],"fun_headline_variants":["Cross-lingual transfer plus 10% span labels gets near-full NER","Match full NER with 6x fewer tokens via targeted span labeling","Bootstrapping NER: cross-lingual start, then label only uncertain spans","Dual strategy: cross-lingual transfer then targeted span annotation for NER"],"cache_read_input_tokens":21632,"weakest_assumption_plain":"The partial-CRF training objective assumes the unannotated tokens in each sequence are missing at random, but ETAL deliberately selects the spans it is most uncertain about, so the training data is a biased sample of the unlabeled corpus.","fun_headline_variants_meta":{"raw":{"variants":["Cross-lingual transfer plus 10% span labels gets near-full NER","Match full NER with 6x fewer tokens via targeted span labeling","Bootstrapping NER: cross-lingual start, then label only uncertain spans","Dual strategy: cross-lingual transfer then targeted span annotation for NER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000674,"raw_usage":{"total_tokens":3057,"prompt_tokens":923,"completion_tokens":2134,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2048}},"tokens_in":539,"tokens_out":2134,"duration_ms":15823,"temperature":1.0,"reasoning_tokens":2048,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:25:10.724779+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ETAL's spans but weight the partial-CRF likelihood with the inverse probability of span selection, or compare ETAL against a random span sampler that is forced to select the same number and length of entities; if the F1 advantage shrinks or disappears, the reported 9.9 F1 gain is an artifact of selection bias rather than of targeting informative entities.","supporting_citations":[{"cited_title":"Smith, and Jaime Carbonell","cited_arxiv_id":null,"evidence_quote":"Supplies the cross-lingual transfer model and resources that initialize the NER model from English training data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the confidence-field-estimation baseline (CFEAL) that selects least confident spans."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the sequence-level active learning baseline (SAL) that selects least confident full sentences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes partial and constrained CRF training on incompletely annotated data, used as PARTIAL-CRF."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the constrained CRF decoder that marginalizes over unannotated tokens."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the paired bootstrap resampling method used to test statistical significance."}],"review_version":1}