Pith. sign in

REVIEW 4 major objections 5 minor 36 references

Query-Based Named Entity Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that recasting named entity recognition as a question-answering task — asking "Which person is mentioned in the text?" for each entity type — solves the long-standing problem of overlapping or nested entities and sets new…

desk verdict A promising NER-as-MRC framing with consistent reported gains, but the published algorithm as written returns one span per query and cannot reproduce the claimed multi-span results. read the letter →

arxiv 1908.09138 v2 pith:4HEEQQYN submitted 2019-08-24 cs.CL

Yuxian Meng , Xiaoya Li , Zijun Sun , Jiwei Li This is my paper · ORCID
classification cs.CL
keywords namedentityrecognitionmachinereadingcomprehensionquery-basedextractionBERTdicelossnestedentitiesoverlappingspan
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that named entity recognition (NER) can be reformulated as a machine reading comprehension problem: instead of labeling each token, the model answers a natural-language question for each entity type, such as 'Which person is mentioned in the text?' The authors show that a BERT-based question-answering model trained this way outperforms standard sequence-labeling BERT on five English and Chinese NER datasets, including ACE 2004/2005 where entities can be nested or overlapping. The key claim is that the query-based formulation resolves the long-standing difficulty of overlapping/nested entities, because each entity type is extracted by an independent question. The paper also finds that natural-language queries outperform index or single-word queries, and that dice loss helps more than cross-entropy for this span prediction setting.

What carries the argument

The central object is the per-type natural-language query combined with a BERT-based span predictor. Given the query $q_y$ and sentence $X$, the model produces, for each token, three scores: start, end, or neither. A single span is then selected (with the smallest start and largest end among predicted positions, or NULL if none), and that span is labeled with $y$. The query templates (e.g., 'Which facility is mentioned in the text?') carry the prior knowledge, and dice loss handles the extreme class imbalance between the one or two positive positions and all other tokens. This machinery replaces the conventional CRF/sequence-labeling head with a QA head, allowing the model to inherit advances from machine reading comprehension.

What would settle it

Take a sentence with two distinct person names such as 'Alice and Bob left.' Under the described post-processing, the model would collapse multiple predicted start positions to the smallest and multiple end positions to the largest, yielding a single span covering both names; if the claimed complete extraction of overlapping/nested entities is correct, the system should instead return two PER spans. Checking this on the released model would settle the claim.

Watch

Extended reading notes

Core claim

The central discovery is that NER performance improves when the task is re-framed from token-level classification to answer-span extraction: for each entity type $y$, a hand-written question $q_y$ is concatenated with the sentence, and a BERT model (fine-tuned with dice loss) predicts whether each token is a start, an end, or neither. Iterating over all entity types and assigning the corresponding label to the returned span yields the final annotation. The authors report that this procedure sets new state-of-the-art F1 scores on MSRA (95.75), Resume (96.97), Chinese OntoNotes (82.11), ACE 2004 (84.14), and ACE 2005 (86.88), beating fine-tuned BERT taggers by 0.14–6.47 points. The claimed mechanism for the improvement is that the query supplies semantic prior knowledge about the target entity class, and that the approach naturally handles overlapping or nested mentions because different entity types are answered by independent questions.

Load-bearing premise

The method assumes that asking one question per entity type returns every mention of that type in the sentence, but the post-processing described in the paper actually selects only one span per question, which would miss multiple separate mentions of the same type.

Editorial extensions

If this is right

  • If the central claim holds, any improvement in MRC span extraction (better readers, better pretraining) transfers directly to NER by simply changing the question templates.
  • The query-based formulation should make NER more data-efficient; the paper's ablation shows it matches full-data BERT with half the training data on OntoNotes.
  • Nested and overlapping entity datasets like ACE 2004/2005 show the largest gains (+4.46 and +6.47 F1 over the BERT tagger), consistent with the claim that the formalization solves the overlap problem.
  • Dice loss contributes consistent gains over cross-entropy across the three datasets tested, suggesting the three-way start/end/neither formulation is particularly imbalance-prone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not explore is zero-shot or few-shot NER: since the query supplies the semantic definition of the entity type, a model might recognize a new entity category from a paraphrase of its query without retraining.
  • The same 'one question per category' recipe could be applied to other span-level tasks, such as event extraction or aspect-based sentiment analysis, where overlapping spans across categories are common.
  • One detail in the paper's post-processing (keeping only the smallest start and largest end per query) implies the system may return a single span per entity type per sentence; whether the reported SOTA numbers are affected by this is not addressed, but it suggests a testable boundary on the claimed overlap-handling capability.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. Summary: This paper proposes reformulating named entity recognition as query-based machine reading comprehension. Each entity type is associated with a natural-language question, and a BERT-based MRC model predicts start and end indices in the input sentence; the predicted span is then assigned the corresponding entity label. The authors claim that this formulation solves overlapping and nested entity handling and achieves state-of-the-art F1 on MSRA, Resume, Chinese OntoNotes, ACE04, and ACE05, with gains over a fine-tuned BERT tagger of 0.14 to 6.47 points. The paper also includes ablations on training data size, query type, and loss function.

Significance. Significance: If the reported results are reproducible, this is an influential reformulation of NER: it leverages pretrained MRC models, encodes type information through queries, and provides a natural way to assign multiple labels to the same token. The consistent gains over the BERT tagger across flat and nested datasets and the query-type ablation are valuable empirical observations. The paper is not fully self-contained, however, and its central claim depends on a decoding procedure that is not described in the manuscript; the significance is therefore conditional on the authors specifying and validating a multi-span decoding scheme.

major comments (4)
  1. [Section 3.2, Algorithm 1, Footnote 2] The decoding procedure described in the manuscript cannot output the labels needed for the reported results. The footnote states that if multiple start positions are predicted, only the smallest index is kept, and if multiple end positions are predicted, only the largest index is kept; Algorithm 1 then writes a single interval into the label sequence for each query. For a sentence containing two mentions of the same type, such as "John and Mary came", this procedure produces at most one span of that type, and the smallest-start/largest-end rule can merge the two mentions into the invalid span "John and Mary". This is inconsistent with the abstract's claim that the model solves overlapping or nested entities and with the reported ACE05 F1 of 86.88. The authors must specify the actual multi-span decoding algorithm, or state and justify a restriction to sentences with at most one mention per type, and adjust the claims accordingly.
  2. [Section 3.3] The sentence "there is at most one token labeled as starting or ending" is not true for gold NER data whenever a sentence contains multiple mentions of the same entity type. This assumption is load-bearing because it defines the positive/negative balance that the dice loss is designed to address and determines how training examples are constructed from the gold labels. If the training procedure actually uses all gold spans for a query, the loss and the data-imbalance argument must be presented with multiple positive starts and ends; if it uses only one span per query, the evaluation should be restricted accordingly, which would not match standard NER benchmarks.
  3. [Section 3.3, Eq. (1)] The two displayed formulas for dice loss are not equivalent. The first line can be read as L_dice = 1 - (2*sum(p_i g_i) + lambda) / (sum(p_i^2) + sum(g_i^2) + lambda), while the immediately following line is L_dice = 1 - 2*sum(p_i g_i) / (sum(p_i^2) + sum(g_i^2) + lambda) - lambda*sum(p_i^2) / (sum(p_i^2) + sum(g_i^2) + lambda). The lambda term in the numerator differs (lambda vs. lambda*sum(p_i^2)), so the training objective is not uniquely defined. Please give one consistent formula and state the role of lambda explicitly.
  4. [Section 4] The central empirical claim of new state-of-the-art results rests on single runs without error bars or repeated trials, and the experimental configuration is incompletely reported: learning rates, dropout, batch sizes, exact query templates for all eight Resume types and all ACE types, and the threshold or criterion for returning NULL are not given. The reported gains over the BERT tagger range from 0.14 to 6.47 F1; without variance estimates or full hyperparameter specifications, the reader cannot assess whether the claimed improvements are meaningful or reproducible.
minor comments (5)
  1. [Section 5.1] Figure 1 is referenced in the text, but no figure is included in the manuscript; the claim about performance with less training data cannot be verified from the submitted version.
  2. [Table 3] Table 3 is captioned "Query Type Samples", but the table reports a loss-function comparison; the caption should be corrected.
  3. [Throughout] There are numerous typos and formatting errors, including "wehre" in Eq. (1), "pertaining" for "pretraining", "mmodels" in Section 4.2, "gargh" for "graph", and "facilitatew" in Section 1; a careful proofread is needed.
  4. [Algorithm 1] Algorithm 1 uses the notation "{ystart,...,yend} = y", which is unclear; the assignment of labels to the predicted span should be written explicitly.
  5. [Footnote 1] Footnote 1 defers the full method to another arXiv paper; for a standalone submission, the decoding and training details must be described in the present paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the SOTA claim is an empirical result against external benchmarks, not a derivation from fitted inputs.

full rationale

The paper's central claim—that query-based MRC with one question per entity type achieves state-of-the-art NER—is an empirical result. The reported F1 scores are computed on held-out test splits of MSRA, Resume, Chinese OntoNotes, ACE04, and ACE05 against external baselines; no equation in the paper is defined in terms of the quantity it is said to predict. The natural-language queries are hand-written templates, and the tuning of hyperparameters (learning rate, dropout, batch size, and the dice-loss λ) is carried out on development data in the standard way; the test results are not a relabeled fit. The footnote referring to the authors' full version is a pointer to additional detail and is not used to justify the SOTA claim, so it is not load-bearing. The apparent inconsistency between Algorithm 1's single-span output and the need to extract multiple same-type mentions is a reproducibility/correctness concern, not circularity: it does not make the reported result equivalent to an input by construction. No self-definitional, fitted-input-as-prediction, or authors-imported-uniqueness pattern is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper provides no mathematical derivation; its claims rest on empirical transfer assumptions. The query templates are hand-selected inputs, and the dice loss lambda is an undisclosed tuned value. Algorithm 1's label assignment assumes per-type spans do not conflict, which is the weakest part of the ledger.

free parameters (3)
  • Dice loss regularization lambda = not reported
    Equation (1) introduces lambda to trade precision and recall; the paper reports no value and says hyperparameters are tuned on the development set (Sections 3.3, 4.1).
  • Training hyperparameters (learning rate, dropout, batch size) = not reported
    Section 4.1 says all are tuned by grid search on the development set, but values are not given.
  • Manual query templates = natural-language templates per entity type (e.g., 'Which facility is mentioned in the text?')
    Section 3.2 and Table 1 show templates chosen by hand; the comparison in Table 4 suggests query wording changes F1, so this is a hand-chosen component the claim depends on.
assumptions (4)
  • domain assumption BERT's pretrained representations transfer to span-based NER and encode enough semantics to answer entity-type questions.
    Section 3.2 uses BERT as the MRC backbone without additional justification beyond its SQuAD performance; the transfer is assumed.
  • domain assumption Natural-language query wording provides useful prior knowledge for entity extraction.
    Section 1 and Table 4 rely on this semantic prior; it is an empirical assumption tested only indirectly.
  • domain assumption Dice loss better handles the extreme class imbalance of start/end positions than cross-entropy.
    Section 3.3 asserts this and Table 3 compares on three datasets, but there is no formal argument.
  • ad hoc to paper The final label sequence can be produced by assigning each extracted span one label, one type at a time.
    Algorithm 1 iterates over questions and overwrites labels; this assumes spans from different queries do not conflict, which is not proven and conflicts with the nested-entity claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Query-Based Named Entity Recognition." pith.science (2026). https://pith.science/paper/4HEEQQYN

@misc{pith2026190809138,
  author       = {Pith},
  title        = {Pith review of: Query-Based Named Entity Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4HEEQQYN}},
  note         = {Machine review of arXiv:1908.09138}
}
read the original abstract

In this paper, we propose a new strategy for the task of named entity recognition (NER). We cast the task as a query-based machine reading comprehension task: e.g., the task of extracting entities with PER is formalized as answering the question of "which person is mentioned in the text ?". Such a strategy comes with the advantage that it solves the long-standing issue of handling overlapping or nested entities (the same token that participates in more than one entity categories) with sequence-labeling techniques for NER. Additionally, since the query encodes informative prior knowledge, this strategy facilitates the process of entity extraction, leading to better performances. We experiment the proposed model on five widely used NER datasets on English and Chinese, including MSRA, Resume, OntoNotes, ACE04 and ACE05. The proposed model sets new SOTA results on all of these datasets.

Figures

Figures reproduced from arXiv: 1908.09138 by the authors.

Figure 1
Figure 1. model performance decrease with less train [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 19 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051

    arXiv 2017

  4. [4]

    Jason PC Chiu and Eric Nichols. 2016 a . Named entity recognition with bidirectional lstm-cnns. Transactions of the Association for Computational Linguistics, 4:357--370

    work page 2016

  5. [5]

    Jason PC Chiu and Eric Nichols. 2016 b . Named entity recognition with bidirectional lstm-cnns. Transactions of the Association for Computational Linguistics, 4:357--370

    work page 2016

  6. [6]

    Ronan Collobert, Jason Weston, L \'e on Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493--2537

    2011

  7. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

  8. [8]

    George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie M Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In Lrec, volume 2, page 1

    work page 2004

Show all 36 references
  1. [9]

    Jenny Rose Finkel and Christopher D Manning. 2009. Nested named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pages 141--150. Association for Computational Linguistics

    2009

  2. [10]

    James Hammerton. 2003. Named entity recognition with long short-term memory. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 172--175. Association for Computational Linguistics

    2003

  3. [11]

    Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 861--871

    2018

  4. [12]

    J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003. Genia corpus—a semantically annotated corpus for bio-textmining. Bioinformatics, 19(suppl\_1):i180--i182

    2003

  5. [13]

    John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data

    2001

  6. [14]

    Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360

    2016 arXiv

  7. [15]

    Gina - Anne Levow. 2006. https://aclanthology.info/papers/W06-0115/w06-0115 The third international chinese language processing bakeoff: Word segmentation and named entity recognition . In Proceedings of the Fifth Workshop on Chinese Language Processing, SIGHAN@COLING/ACL 2006...

    2006

  8. [16]

    Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv:1706.04115

    2017 arXiv

  9. [17]

    Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 857--867

    2015

  10. [18]

    Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354

    2016 arXiv

  11. [19]

    Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730

    2018 arXiv

  12. [20]

    Nassir Navab Milletari, Fausto and Seyeds-Ahmad Ahmadi. 2016. V-net: Fully convolutional neural networks for volumetric medical image segmentation. 2016 Fourth International Conference on 3D Vision (3DV)

    2016

  13. [21]

    Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365

    2018 arXiv

  14. [22]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250

    2016 arXiv

  15. [23]

    Cicero Nogueira dos Santos and Victor Guimaraes. 2015. Boosting named entity recognition with neural character embeddings. arXiv preprint arXiv:1505.05008

    2015 arXiv

  16. [24]

    Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603

    2016 arXiv

  17. [25]

    Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1047--1055. ACM

    2017

  18. [26]

    Charles Sutton, Andrew McCallum, and Khashayar Rohanimanesh. 2007. Dynamic conditional random fields: Factorized probabilistic models for labeling and segmenting sequence data. Journal of Machine Learning Research, 8(Mar):693--723

    2007

  19. [27]

    Bailin Wang and Wei Lu. 2018. Neural segmental hypergraphs for overlapping mention recognition. arXiv preprint arXiv:1810.01817

    2018 arXiv

  20. [28]

    Bailin Wang, Wei Lu, Yu Wang, and Hongxia Jin. 2018. A neural transition-based model for nested mention recognition. arXiv preprint arXiv:1810.01808

    2018 arXiv

  21. [29]

    Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. arXiv preprint arXiv:1608.07905

    2016 arXiv

  22. [30]

    Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2016. Multi-perspective context matching for machine comprehension. arXiv preprint arXiv:1612.04211

    2016 arXiv

  23. [31]

    Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, Robert Belvin, et al. 2011. Ontonotes release 4.0. LDC2011T03, Philadelphia, Penn.: Linguistic Data Consortium

    2011

  24. [32]

    Wei Wu, Yuxian Meng, Qinghong Han, Muyu Li, Xiaoya Li, Jie Mei, Ping Nie, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for chinese character representations. arXiv preprint arXiv:1901.10125

    2019 arXiv

  25. [33]

    Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604

    2016 arXiv

  26. [34]

    Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dcn+: Mixed objective and deep residual coattention for question answering. arXiv preprint arXiv:1711.00106

    2017 arXiv

  27. [35]

    Mingbin Xu, Hui Jiang, and Sedtawut Watcharawittayakul. 2017. A local detection approach for named entity recognition and mention detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1237--1247

    2017

  28. [36]

    Yue Zhang and Jie Yang. 2018. Chinese ner using lattice lstm. arXiv preprint arXiv:1805.02023

    2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.