{"id":"fa7475a2-79e1-45af-8418-38544ce71af1","arxiv_id":"1908.09138","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Named entity recognition can be reformulated as answering one natural-language question per entity type with a BERT span extractor, and the paper reports state-of-the-art results on five datasets.","lead":"This paper reframes named entity recognition as a question-answering task, where each entity type is a natural-language question and a BERT-based reader extracts answer spans. The authors report state-of-the-art F1 scores on five English and Chinese datasets and claim this formulation handles nested entities.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 returns one span per query, so the described system cannot label multiple mentions of a type; the reported SOTA requires an unspecified multi-span decoding.","rationale":"The reader's weakest assumption correctly identifies the one-span-per-query issue. My review confirms and sharpens it: the problem is not only an ambiguous footnote but a contradiction with the training objective in §3.3, which assumes at most one start and one end per query while NER corpora routinely contain multiple same-type mentions. Since every headline result (SOTA on five datasets, solution to nested entities) depends on the decoder emitting the complete set of spans, this is the single load-bearing weak point. The reader's CONDITIONAL verdict is appropriate: the full version/code may contain the missing multi-span decoding, but as submitted the method is under-specified and the numbers cannot be derived from the description. I do not move the verdict because the same conditional gate is already stated; the contribution is plausible and the reported pattern of gains is consistent, but verification requires the full-version decoding details or code.","tokens_in":6976,"tokens_out":6525,"duration_ms":73058,"concrete_test":"Run the exact inference pipeline described by Algorithm 1 and the §3.2 post-processing rule on ACE05 using the authors' BERT Query setup: for every test sentence containing k mentions of the same type, record how many spans of that type are emitted per query. If the decoder emits fewer than k spans on any such sentence, the published method cannot reproduce the reported 86.88 F1; checking the full version (arXiv:1910.11476) or released code for a multi-span pairing step would settle whether a different decode was actually used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that one MRC query per entity type extracts all entities and solves nested/overlapping NER depends on the decoder emitting every span of that type from a single query. Section 3.2's footnote specifies the opposite: if multiple start positions are predicted, keep only the smallest index; if multiple end positions are predicted, keep only the largest index. Algorithm 1 then overwrites labels with one interval per query. Thus in a sentence containing two PER mentions, at most one PER span can be produced, and the smallest-start/largest-end rule can even merge the two mentions into one invalid span. Section 3.3 states 'there is at most one token labeled as starting or ending' for a given query, which is false for gold NER data whenever a sentence has multiple same-type entities. These two statements are internally inconsistent with the reported F1 scores (e.g., ACE05 86.88). The results can only be real if the actual system uses a multi-span decoding scheme that is absent from this paper; without that, the SOTA claims are not reproducible from the published method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Summary: This paper proposes reformulating named entity recognition as query-based machine reading comprehension. Each entity type is associated with a natural-language question, and a BERT-based MRC model predicts start and end indices in the input sentence; the predicted span is then assigned the corresponding entity label. The authors claim that this formulation solves overlapping and nested entity handling and achieves state-of-the-art F1 on MSRA, Resume, Chinese OntoNotes, ACE04, and ACE05, with gains over a fine-tuned BERT tagger of 0.14 to 6.47 points. The paper also includes ablations on training data size, query type, and loss function.","tokens_in":7163,"tokens_out":6694,"duration_ms":66437,"significance":"Significance: If the reported results are reproducible, this is an influential reformulation of NER: it leverages pretrained MRC models, encodes type information through queries, and provides a natural way to assign multiple labels to the same token. The consistent gains over the BERT tagger across flat and nested datasets and the query-type ablation are valuable empirical observations. The paper is not fully self-contained, however, and its central claim depends on a decoding procedure that is not described in the manuscript; the significance is therefore conditional on the authors specifying and validating a multi-span decoding scheme.","major_comments":[{"comment":"The decoding procedure described in the manuscript cannot output the labels needed for the reported results. The footnote states that if multiple start positions are predicted, only the smallest index is kept, and if multiple end positions are predicted, only the largest index is kept; Algorithm 1 then writes a single interval into the label sequence for each query. For a sentence containing two mentions of the same type, such as \"John and Mary came\", this procedure produces at most one span of that type, and the smallest-start/largest-end rule can merge the two mentions into the invalid span \"John and Mary\". This is inconsistent with the abstract's claim that the model solves overlapping or nested entities and with the reported ACE05 F1 of 86.88. The authors must specify the actual multi-span decoding algorithm, or state and justify a restriction to sentences with at most one mention per type, and adjust the claims accordingly.","section":"Section 3.2, Algorithm 1, Footnote 2"},{"comment":"The sentence \"there is at most one token labeled as starting or ending\" is not true for gold NER data whenever a sentence contains multiple mentions of the same entity type. This assumption is load-bearing because it defines the positive/negative balance that the dice loss is designed to address and determines how training examples are constructed from the gold labels. If the training procedure actually uses all gold spans for a query, the loss and the data-imbalance argument must be presented with multiple positive starts and ends; if it uses only one span per query, the evaluation should be restricted accordingly, which would not match standard NER benchmarks.","section":"Section 3.3"},{"comment":"The two displayed formulas for dice loss are not equivalent. The first line can be read as L_dice = 1 - (2*sum(p_i g_i) + lambda) / (sum(p_i^2) + sum(g_i^2) + lambda), while the immediately following line is L_dice = 1 - 2*sum(p_i g_i) / (sum(p_i^2) + sum(g_i^2) + lambda) - lambda*sum(p_i^2) / (sum(p_i^2) + sum(g_i^2) + lambda). The lambda term in the numerator differs (lambda vs. lambda*sum(p_i^2)), so the training objective is not uniquely defined. Please give one consistent formula and state the role of lambda explicitly.","section":"Section 3.3, Eq. (1)"},{"comment":"The central empirical claim of new state-of-the-art results rests on single runs without error bars or repeated trials, and the experimental configuration is incompletely reported: learning rates, dropout, batch sizes, exact query templates for all eight Resume types and all ACE types, and the threshold or criterion for returning NULL are not given. The reported gains over the BERT tagger range from 0.14 to 6.47 F1; without variance estimates or full hyperparameter specifications, the reader cannot assess whether the claimed improvements are meaningful or reproducible.","section":"Section 4"}],"minor_comments":[{"comment":"Figure 1 is referenced in the text, but no figure is included in the manuscript; the claim about performance with less training data cannot be verified from the submitted version.","section":"Section 5.1"},{"comment":"Table 3 is captioned \"Query Type Samples\", but the table reports a loss-function comparison; the caption should be corrected.","section":"Table 3"},{"comment":"There are numerous typos and formatting errors, including \"wehre\" in Eq. (1), \"pertaining\" for \"pretraining\", \"mmodels\" in Section 4.2, \"gargh\" for \"graph\", and \"facilitatew\" in Section 1; a careful proofread is needed.","section":"Throughout"},{"comment":"Algorithm 1 uses the notation \"{ystart,...,yend} = y\", which is unclear; the assignment of labels to the predicted span should be written explicitly.","section":"Algorithm 1"},{"comment":"Footnote 1 defers the full method to another arXiv paper; for a standalone submission, the decoding and training details must be described in the present paper.","section":"Footnote 1"}],"recommendation":"major_revision","confidential_remarks":"This manuscript appears to be a shortened version of arXiv:1910.11476 and defers key details to that paper. I recommend asking the authors to make the submission self-contained and to confirm that the full version's decoding procedure matches what is described here. The internal inconsistency in Section 3.2 is serious enough that I would not recommend acceptance without seeing the corrected method and ideally a small verification experiment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core idea—one natural-language query per entity type, answered by a BERT MRC model—is a genuinely useful way to express NER, and it does sidestep the BIO-tagging issue for overlapping mentions. Second, the paper as written does not actually describe a decoder that can extract multiple spans of the same type from one query, so the headline results are not reproducible from the method given.\n\nWhat is new: applying the QA-as-task trick to NER with per-type templates, plus the observation that natural language queries beat index or single-word pseudo queries. The comparison against a BERT tagger is the right baseline, and the gains are consistent in direction across all five datasets. The dice-loss ablation is also sensible, and the paper is honest about related work: Levy et al. and McCann et al. already cast other tasks as QA.\n\nThe soft spot is load-bearing. Footnote 2 in Section 3.2 says that if multiple start positions are predicted, keep the smallest index, and if multiple ends are predicted, keep the largest. That yields one span per query. Section 3.3 says there is at most one token labeled as starting or ending for a given query, which is false for ordinary NER data whenever a sentence has two mentions of the same type. Algorithm 1 then overwrites labels with one interval per question. Such a system could never output two PER spans from a single sentence, yet ACE05 has frequent same-type mentions and the paper reports F1 86.88. Unless the real system uses a multi-span decoding scheme not present in this text, the results cannot be produced by the described method. No code is provided, and there are no error bars or repeated runs, but those are secondary.\n\nI don't think the numbers are fabricated; the full version (arXiv:1910.11476) may spell out the actual decoding. But referees cannot verify the central claim from this manuscript. The citation pattern is fine—the self-citation is to the extended version of the same work, not a dodge.\n\nRecommendation: this deserves a serious referee. The framing is valuable and the results, if real, matter to the NER community. Send it out, but it should be accepted only after the authors disclose the multi-span decoding algorithm, release code, and ideally add variance over runs. As is, it is a conditional at best.","headline":"A promising NER-as-MRC framing with consistent reported gains, but the published algorithm as written returns one span per query and cannot reproduce the claimed multi-span results.","tokens_in":7689,"tokens_out":3158,"would_cite":false,"duration_ms":31622,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that recasting named entity recognition as a question-answering task — asking \"Which person is mentioned in the text?\" for each entity type — solves the long-standing problem of overlapping or nested entities and sets new…","keywords":["named entity recognition","machine reading comprehension","query-based extraction","BERT","dice loss","nested entities","overlapping entities","span extraction"],"falsifier":"Take a sentence with two distinct person names such as 'Alice and Bob left.' Under the described post-processing, the model would collapse multiple predicted start positions to the smallest and multiple end positions to the largest, yielding a single span covering both names; if the claimed complete extraction of overlapping/nested entities is correct, the system should instead return two PER spans. Checking this on the released model would settle the claim.","tokens_in":6752,"feed_emoji":"❓","tokens_out":4741,"duration_ms":44042,"temperature":0.7,"pith_summary":"This paper argues that named entity recognition (NER) can be reformulated as a machine reading comprehension problem: instead of labeling each token, the model answers a natural-language question for each entity type, such as 'Which person is mentioned in the text?' The authors show that a BERT-based question-answering model trained this way outperforms standard sequence-labeling BERT on five English and Chinese NER datasets, including ACE 2004/2005 where entities can be nested or overlapping. The key claim is that the query-based formulation resolves the long-standing difficulty of overlapping/nested entities, because each entity type is extracted by an independent question. The paper also finds that natural-language queries outperform index or single-word queries, and that dice loss helps more than cross-entropy for this span prediction setting.","feed_headline":"Asking 'who is mentioned?' beats tagging for entity extraction","feed_subtitle":"A BERT model answers one question per entity type and sets new state-of-the-art F1 scores on five NER datasets.","key_machinery":"The central object is the per-type natural-language query combined with a BERT-based span predictor. Given the query $q_y$ and sentence $X$, the model produces, for each token, three scores: start, end, or neither. A single span is then selected (with the smallest start and largest end among predicted positions, or NULL if none), and that span is labeled with $y$. The query templates (e.g., 'Which facility is mentioned in the text?') carry the prior knowledge, and dice loss handles the extreme class imbalance between the one or two positive positions and all other tokens. This machinery replaces the conventional CRF/sequence-labeling head with a QA head, allowing the model to inherit advances from machine reading comprehension.","core_discovery":"The central discovery is that NER performance improves when the task is re-framed from token-level classification to answer-span extraction: for each entity type $y$, a hand-written question $q_y$ is concatenated with the sentence, and a BERT model (fine-tuned with dice loss) predicts whether each token is a start, an end, or neither. Iterating over all entity types and assigning the corresponding label to the returned span yields the final annotation. The authors report that this procedure sets new state-of-the-art F1 scores on MSRA (95.75), Resume (96.97), Chinese OntoNotes (82.11), ACE 2004 (84.14), and ACE 2005 (86.88), beating fine-tuned BERT taggers by 0.14–6.47 points. The claimed mechanism for the improvement is that the query supplies semantic prior knowledge about the target entity class, and that the approach naturally handles overlapping or nested mentions because different entity types are answered by independent questions.","pith_inferences":["A natural extension the paper does not explore is zero-shot or few-shot NER: since the query supplies the semantic definition of the entity type, a model might recognize a new entity category from a paraphrase of its query without retraining.","The same 'one question per category' recipe could be applied to other span-level tasks, such as event extraction or aspect-based sentiment analysis, where overlapping spans across categories are common.","One detail in the paper's post-processing (keeping only the smallest start and largest end per query) implies the system may return a single span per entity type per sentence; whether the reported SOTA numbers are affected by this is not addressed, but it suggests a testable boundary on the claimed overlap-handling capability."],"forward_implications":["If the central claim holds, any improvement in MRC span extraction (better readers, better pretraining) transfers directly to NER by simply changing the question templates.","The query-based formulation should make NER more data-efficient; the paper's ablation shows it matches full-data BERT with half the training data on OntoNotes.","Nested and overlapping entity datasets like ACE 2004/2005 show the largest gains (+4.46 and +6.47 F1 over the BERT tagger), consistent with the claim that the formalization solves the overlap problem.","Dice loss contributes consistent gains over cross-entropy across the three datasets tested, suggesting the three-way start/end/neither formulation is particularly imbalance-prone."],"supporting_citations":[{"why":"Supplies the BERT backbone that the query-based model is built on and fine-tuned from.","marker":"(Devlin et al., 2018)"},{"why":"Provides the dice loss objective that replaces cross-entropy for span-start/end prediction.","marker":"(Milletari and Ahmadi, 2016)"},{"why":"Establishes the span-extraction strategy (predicting start and end positions) that the model adapts.","marker":"(Seo et al., 2016)"},{"why":"Shows how a relation extraction task can be transformed into question answering, a direct precursor to the query-based NER idea.","marker":"(Levy et al., 2017)"},{"why":"Provides the MSRA Chinese NER benchmark used in the experiments.","marker":"(Levow, 2006)"},{"why":"Supplies the Resume dataset and the Lattice LSTM baseline system that the query model must beat.","marker":"(Zhang and Yang, 2018)"},{"why":"Supplies Chinese OntoNotes 4.0, one of the five test datasets.","marker":"(Weischedel et al., 2011)"},{"why":"Provides the ACE 2004/2005 datasets that contain nested and overlapping entities, the core testbed for the paper's main claimed advantage.","marker":"(Doddington et al., 2004)"}],"fun_headline_variants":["Query-based NER: ask 'who?' per type, set SOTA on five benchmarks","Turn NER into question answering: handles nested, beats tagging","Ask BERT one question per entity type to claim new SOTA","Forget token tags: ask 'which person?' to extract entities","NER as QA: per-type questions ace overlapping and nested spans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that asking one question per entity type returns every mention of that type in the sentence, but the post-processing described in the paper actually selects only one span per question, which would miss multiple separate mentions of the same type.","fun_headline_variants_meta":{"raw":{"variants":["Query-based NER: ask 'who?' per type, set SOTA on five benchmarks","Turn NER into question answering: handles nested, beats tagging","Ask BERT one question per entity type to claim new SOTA","Forget token tags: ask 'which person?' to extract entities","NER as QA: per-type questions ace overlapping and nested spans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000342,"raw_usage":{"total_tokens":1855,"prompt_tokens":891,"completion_tokens":964,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":870}},"tokens_in":507,"tokens_out":964,"duration_ms":10195,"temperature":1.0,"reasoning_tokens":870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:20:47.664776+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sentence with two distinct person names such as 'Alice and Bob left.' Under the described post-processing, the model would collapse multiple predicted start positions to the smallest and multiple end positions to the largest, yielding a single span covering both names; if the claimed complete extraction of overlapping/nested entities is correct, the system should instead return two PER spans. Checking this on the released model would settle the claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the dice loss objective that replaces cross-entropy for span-start/end prediction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MSRA Chinese NER benchmark used in the experiments."},{"cited_title":"Chinese NER Using Lattice LSTM","cited_arxiv_id":"1805.02023","evidence_quote":"Supplies the Resume dataset and the Lattice LSTM baseline system that the query model must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ACE 2004/2005 datasets that contain nested and overlapping entities, the core testbed for the paper's main claimed advantage."}],"review_version":1}