Position-Aware Self-Attention based Neural Sequence Labeling

Guangyou Zhou; Pan Zhou; Sheng Jiang; Wei Wei; Xianling Mao; Zanbo Wang

REVIEW 3 major objections 6 minor 48 references

Position-Aware Self-Attention based Neural Sequence Labeling

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding a position-aware self-attention layer with three positional biases to a Bi-LSTM-CRF model improves sequence labeling enough to beat the previous state of the art on NER, POS tagging, and chunking without…

desk verdict Competent attention extension with clear ablations, but the headline F1 is compromised by a likely test-set hyperparameter search and tiny, significance-free gains. read the letter →

arxiv 1908.09128 v2 pith:OKGDA2P2 submitted 2019-08-24 cs.CL cs.LG

Wei Wei , Zanbo Wang , Xianling Mao , Guangyou Zhou , Pan Zhou , Sheng Jiang This is my paper · ORCID

classification cs.CLcs.LG

keywords sequencelabelingposition-awareself-attentiondiscretecontextdependenciesnamedentityrecognitionpart-of-speechtaggingchunkingBi-LSTM-CRFpositionalbias

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that standard Bi-LSTM-CRF sequence labeling misses non-neighboring, discrete dependencies between words, and it proposes a position-aware self-attention (PSA) mechanism to recover them. PSA adds three positional terms to the additive attention score: a self-disabled mask, a distance-aware Gaussian bias, and a token-specific relative-position bias. A self-attentional context fusion layer with a learned gate combines the attention output with the original token representation. On CoNLL 2003 NER, WSJ POS tagging, and CoNLL 2000 chunking, the model is reported to outperform earlier systems that do not use external knowledge, with average F1 of 91.33 on NER, accuracy of 97.59 on POS tagging, and F1 of 95.09 on chunking. The paper's thesis is that discrete context dependencies are real and that a comparatively simple attention extension can capture them.

What carries the argument

The load-bearing object is the position-aware self-attention (PSA) layer, an extension of additive self-attention in which a positional bias function $\Psi_{ij}(\hat{x}_i)$ is added to the pairwise compatibility score. The function combines a diagonal mask that stops a token from attending to itself, a Gaussian penalty that down-weights distant neighbors by $(i-j)^2$, and a learned relative-position term that scales the focused token's representation by a position-dependent row of a weight matrix. The companion self-attentional context fusion layer uses a sigmoid gate $\lambda$ to interpolate between the token's original representation and the transformed attention output, which repairs the information removed by the self-disabled mask. These pieces together make the relative position of every token pair explicit at attention time, which the paper claims lets the model capture discrete dependencies such as a verb governing a non-adjacent adverb.

What would settle it

Re-run the full model on CoNLL03 NER with $k$ chosen by validation performance over a grid such as 2, 5, 8, 10, 15, and 20, separately choosing $r$, and report the average over the same five random restarts; if the best validation choice is not $k=10$ or the gap over the Bi-LSTM-CRF baseline of 91.01 average F1 disappears, the paper's central claim is not supported.

Watch

Extended reading notes

Core claim

The central claim is that a Bi-LSTM-CRF tagger can be improved by inserting two self-attentional context fusion layers whose attention scores are made position-aware through three additive biases. The compatibility score becomes $f(\hat{x}_i,\hat{x}_j)=\hat{w}^\top\sigma(W^{(1)}\hat{x}_i+W^{(2)}\hat{x}_j+\hat{b})+\Psi_{ij}(\hat{x}_i)$, where $\Psi$ combines a self-disabled mask $M_{ij}$, a Gaussian distance bias $G_{ij}=-(i-j)^2/(2\varepsilon^2)$, and a token-specific position bias $P_{ij}=\hat{x}_i W^{(3)}_{C(i-j,r)}$. The fusion layer computes a gate $\lambda$ that mixes the original token vector with the attention output, so the disabled self-attention does not lose the token's own information. The three-bias combination is presented as the reason the model outperforms the Bi-LSTM-CRF baseline by 0.32 F1 on NER, 0.08 accuracy on POS tagging, and 0.17 F1 on chunking, and also outperforms prior no-external-knowledge systems.

Load-bearing premise

The reported gains rest on the assumption that the Gaussian window size $k=10$ and its tied clipping threshold $r$ were selected using development data, not test data; if they were tuned on the test set, the headline numbers overstate the true advantage.

Editorial extensions

If this is right

A sequence tagger can gain accuracy by making attention position-aware rather than by adding more RNN layers or external lexicons.
The same three-bias attention can be inserted into other Bi-LSTM-CRF systems as a drop-in context fusion layer, since it only modifies the context encoder.
Because the largest reported gains are on short sentences, discrete short-range dependencies are where the mechanism contributes most.
The fusion gate provides a way to use self-attention even when the attention mask disables self-contributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension is to test whether the token-specific relative-position bias alone, trained as a full embedding table rather than a clipped scalar, reproduces most of the reported gains at lower implementation cost.
The same mechanism could benefit other token-level tasks with non-local cues, such as semantic role labeling or aspect extraction, where a predicate and its arguments are often separated by several words.
Because the gains concentrate on short sentences, the approach may transfer especially well to informal text such as tweets or chat, where local discrete patterns dominate.
The ablation evidence shows that removing any one bias hurts, but it does not identify which weighting of the three biases is optimal; a learned or per-task weighting could be tested.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short take: this is a competent engineering paper that assembles existing ideas into a position-aware self-attention layer on top of Bi-LSTM-CRF, with a gated fusion layer. The architecture is described clearly and the ablation studies are the right kind of evidence. On three standard benchmarks it beats the re-implemented baseline by 0.08–0.32 points, but the claim that it 'outperforms state-of-the-arts' is too strong—on WSJ POS the average ties Zhang et al. at 97.59, and the differences are small enough that run-to-run noise could explain part of them.

The real problem is the window-size selection in Section V-C4 / Figure 5. The text says k 'must be optimized for' and then investigates it on the CoNLL03 NER task, but never says whether that is the development or test set. Since early stopping is otherwise done on dev, a natural reading is that the reported 91.33 F1 came after picking the best-looking k on the test split. The same k=10 is then used for all tasks, and the clipping threshold r for the token-specific bias is tied to k. If that is what happened, the headline NER number is an optimistic upper bound and the entire advantage of the positional biases could shrink under proper validation. To be fair, the paper does not explicitly say 'test set', so this is an ambiguity rather than a proven violation; but the burden is on the authors to clarify, and they don't.

The ablation (Table V) attributes only 0.18 F1 to the full positional bias, which is within a couple of standard deviations of the runs. Suggestive, not convincing. No significance tests are reported. I would want the authors to redo the k search on the development set, report dev-based selection for all tasks, give confidence intervals or paired tests, and soften the SOTA claim. The combination does have a modest novelty—token-specific position bias as an inner product with the focus vector is a reasonable twist on Shaw et al.—and the paper gives enough implementation detail to be reproduced.

Bottom line: worth a serious referee, not a desk reject. It is a legitimate contribution but needs major revision before the empirical claim is credible. I would not cite it in its current form.

Referee Report

3 major / 6 minor

Summary. The paper proposes a position-aware self-attention (PSA) mechanism for neural sequence labeling, built on a Bi-LSTM-CRF backbone. PSA augments additive self-attention with three positional biases: a self-disabled mask, a distance-aware Gaussian bias, and a token-specific position bias, and the architecture also includes a self-attentional context fusion layer. Experiments on CoNLL 2003 NER, WSJ POS tagging, and CoNLL 2000 chunking report improvements over a re-implemented Bi-LSTM-CRF baseline and claim state-of-the-art performance without external knowledge. The paper includes ablation studies, sentence-length analysis, a window-size sensitivity experiment, and qualitative case studies.

Significance. If the empirical claims were fully supported, this would be a modest but useful contribution: a simple attention extension that adds interpretable position modeling to a standard sequence-labeling architecture. The paper's strengths are its clear model description, the use of three standard benchmarks, repeated runs with reported mean and standard deviation, and explicit comparison with a re-implemented baseline. However, the central claim is currently not established. The reported gains over the baseline are small, the ablation attributes most of the effect to components whose differences are within run-to-run noise, and the key hyperparameter k appears to have been selected using test-set results. The proposed mechanism is coherent, but the evidence presented does not support the conclusion that it outperforms the state of the art.

major comments (3)

[Section V-C4 (Impact of Window Size), Figure 5; Section IV-A3] The manuscript states that the window size k 'is clearly a hyperparameter which must be optimized for' and then investigates its influence on the CoNLL03 NER task, but it never states whether the search is performed on the development set or the test set. Because all other reported numbers in the paper are test-set numbers and early stopping is defined on development sets, the natural reading is that k=10 was chosen after inspecting test-set F1. This is post-hoc data selection. The problem is compounded by the fact that k also sets the clipping threshold r for the token-specific position bias (Section IV-A3), and the same k=10 is then transferred to POS tagging and chunking. The reported NER result of 91.33 is therefore an optimistic upper bound unless the authors demonstrate that the same k would be selected using only development data. The authors should report dev-set F1 for all values of k, describe the selection procedure explicitly, and re-evaluate all three tasks with the chosen k.
[Section V-C2, Tables V and VI] The ablation evidence for the central novelty is weak. Removing the positional bias Psi_ij lowers NER F1 from 91.33 to 91.15, a difference of 0.18, while the reported standard deviations are 0.08 and 0.12; these intervals overlap substantially. The factor-wise ablations in Table VI (91.12, 91.07, and 91.19) are similarly within noise of the full model's 91.33. Because the positional bias is the paper's main contribution, the claim that this mechanism 'achieves a significant improvement' is not supported. The authors should provide significance tests, confidence intervals, or a substantially larger number of runs to show that the observed differences are not explained by seed variation.
[Section V-C1, Tables II-IV] The headline claim that the model outperforms the state of the art without external knowledge rests on very small margins: +0.12 F1 over Ma and Hovy (2016) on NER and +0.04 accuracy on POS, while the NER margin over Liu et al. (2018) is +0.09, smaller than that work's reported standard deviation of 0.12. These comparisons are made against published numbers obtained with different embeddings, character encoders, and optimization settings, so they are not matched comparisons. Without statistical tests and re-implemented baselines under identical conditions, the improvements over published systems are not established. At minimum, the paper should either add matched re-implementations of the most competitive baselines or soften the claim to 'comparable to the state of the art.'

minor comments (6)

[Section IV-A3, Eq. (12)] The clipping function is written incorrectly for negative relative positions: as given, C(q,r) returns r when |q|>r, so a very negative q would be mapped to +r instead of being clipped at -r. The intended definition appears to be symmetric clipping, e.g., C(q,r)=max(-r, min(r, q)).
[Section IV-A, Eqs. (7)-(8)] The parameters alpha_1, alpha_2, and alpha_3 are introduced as trade-off weights with sum one, but the paper never states whether they are learned during training or fixed in advance, and if fixed, what values were used. This is needed for reproducibility.
[Section V-C5, Table IX] The case study table is hard to follow because the number of gold tags and predictions does not match the number of tokens in several examples (for instance, Sent1 appears to have 17 tokens but only 15 tags). The alignment should be fixed.
[Figure 5] The window-size sensitivity plot has no error bars or numerical labels, so the reader cannot judge whether the reported differences across values of k are meaningful or within run-to-run variation.
[Section V-B] There is a typo in 'leaning rate' that should read 'learning rate', and the learning-rate schedule eta_t = eta_0/(1+rho*t) should be stated more precisely with respect to epochs or updates.
[General] No code or implementation details are made available; releasing code would materially help reproducibility, especially given the small effect sizes reported.

Circularity Check

1 steps flagged · score 6.0 of 10

Section V-C4 tunes the Gaussian window size k on the CoNLL03 NER task and the same k is reused for all tasks; the reported NER improvement is therefore a selected maximum rather than an independent prediction.

fitted input called prediction [Section V-C4 'Impact of Window Size' and Figure 5; k defined in Section IV-A2 (Eq. 10); r tied to k in Section IV-A3.]
"The window size k (rf. Section IV-A2) is clearly a hyperparameter which must be optimized for, thus we investigate the inﬂuence of the value of k on the CoNLL2003 NER task. We also rerun 5 times with different random initialization and report the average score, which is consistent with our other experiments in this paper. The plot in Figure 5 shows that when assigning the value of k to 10 we do outperform other models substantially. And with other window sizes (except 2) our model performs relatively well and is superior to the Bi-LSTM-CRF baseline ( 91.01%)."

The reported CoNLL03 NER result (Table II: 'Our model avg 91.33±0.08') is the final model's test F1 with k=10. Section V-C4 reports 'the inﬂuence of the value of k on the CoNLL2003 NER task' and states that k=10 'outperform[s] other models substantially'; it never says this search used the development set, whereas the only explicitly dev-based model selection in the paper is early stopping. Thus the headline value is the best point of the k-sweep on the same test set used for evaluation, i.e., a fitted maximum presented as a prediction. The same k also fixes the clipping threshold r for the token-specific position bias, so two of the three positional biases are controlled by the test-fitted value; this k is then reused for POS tagging and chunking.

full rationale

The only circular step I can exhibit from the paper's own text is in Section V-C4. The paper's central claim is the numeric superiority of PSA over state-of-the-art sequence labelers, and for CoNLL03 NER that claim rests on the 91.33 F1 in Table II. That number is produced with k=10, and Section V-C4 describes investigating k on the CoNLL2003 NER task, with k=10 giving the best result; no development-set curve is reported. Since all other reported CoNLL03 NER numbers in the paper are test-set F1 values, the natural and textually supported reading is that the hyperparameter was selected using the test set, making the reported result the maximum of the sweep rather than an independent prediction. This same k sets the clipping threshold r for the token-specific position bias, so the test-selected value controls two of the three positional factors and is then transferred to the other two tasks. The ablation in Table V attributes only about 0.18 F1 to the positional bias, smaller than the reported run-to-run standard deviations, which is consistent with the small gain being an artifact of selection. I found no self-citation chain, no uniqueness theorem imported from the authors, and no definitional equivalence in the model equations: the PSA mechanism and the fusion layer are not defined in terms of the final F1 scores. The circularity is therefore partial and localized to the hyperparameter-based construction of the headline result, corresponding to a score of 6.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the architecture working as described and on the comparability of benchmark numbers. Two free parameters (k and r) were chosen by hand, and the alpha weights in Eq. 8 are not fully specified. The axioms above are the background assumptions that must hold for the reported gains to be meaningful.

free parameters (3)

k (Gaussian window size) = 10
Controls the standard deviation of the distance-aware Gaussian bias (Eq. 10). The paper states k=10 in Section IV-A2 and selects it based on experiments on CoNLL2003 NER (Section V-C4), apparently on the test set.
r (relative position clipping threshold) = 10 (tied to k)
Maximum relative distance used in the token-specific position bias (Eq. 11, 12). The text says it is equal to the window size k (Section IV-A3).
alpha_1, alpha_2, alpha_3 (positional bias weights) = not specified
Trade-off parameters for the three positional biases in Eq. 8. The paper requires they sum to 1 but does not state whether they are learned or manually set; if manually set, they are free parameters.

assumptions (3)

domain assumption The standard splits and evaluation metrics of the three benchmark datasets are comparable across all cited methods.
The paper compares F1 and accuracy values reported by other papers without re-running those methods under identical settings. Small differences in tokenization, tag scheme (this paper uses BIOES for CoNLL03 while some cited works use BIO), or unknown-word handling could shift scores by more than the reported gains.
ad hoc to paper The Gaussian bias with the chosen k=10 transfers to all three tasks.
Section V-C4 selects k based on CoNLL03 NER results and then uses the same value for POS tagging and chunking without a validation study on those tasks.
ad hoc to paper The token-specific position bias clipping threshold r equals the window size k.
Section IV-A3 states the maximum margin is clipped to r and that its value equals the window size k; this coupling is not independently motivated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Position-Aware Self-Attention based Neural Sequence Labeling." pith.science (2026). https://pith.science/paper/OKGDA2P2

@misc{pith2026190809128,
  author       = {Pith},
  title        = {Pith review of: Position-Aware Self-Attention based Neural Sequence Labeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKGDA2P2}},
  note         = {Machine review of arXiv:1908.09128}
}

read the original abstract

Sequence labeling is a fundamental task in natural language processing and has been widely studied. Recently, RNN-based sequence labeling models have increasingly gained attentions. Despite superior performance achieved by learning the long short-term (i.e., successive) dependencies, the way of sequentially processing inputs might limit the ability to capture the non-continuous relations over tokens within a sentence. To tackle the problem, we focus on how to effectively model successive and discrete dependencies of each token for enhancing the sequence labeling performance. Specifically, we propose an innovative attention-based model (called position-aware selfattention, i.e., PSA) as well as a well-designed self-attentional context fusion layer within a neural network architecture, to explore the positional information of an input sequence for capturing the latent relations among tokens. Extensive experiments on three classical tasks in sequence labeling domain, i.e., partof-speech (POS) tagging, named entity recognition (NER) and phrase chunking, demonstrate our proposed model outperforms the state-of-the-arts without any external knowledge, in terms of various metrics.

Figures

Figures reproduced from arXiv: 1908.09128 by the authors.

**Figure 2.** Overview of proposed neural architecture. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Self-Attentional Context Fusion Network. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Performances on Different Lengths. demonstrates the effectiveness of our well designed positional bias to explore the relative position information of tokens from different perspectives. The result also shows that after removing Pij (xˆi), the F1-score decreases the mo…

**Figure 5.** Figure 5: Performance of our model with various window sizes. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Heatmaps of four cases. of (20;40) our model performs slightly better than Bi-LSTMCRF. 4) Impact of Window Size: The window size k (rf. Section IV-A2) is clearly a hyperparameter which must be optimized for, thus we investigate the influence of the value of k on the C…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 44 canonical work pages

[1]

End-to-end sequence labeling via bi-directional lstm-cnns-crf,

X. Ma and E. Hovy, “End-to-end sequence labeling via bi-directional lstm-cnns-crf,” in ACL, 2016

work page 2016
[2]

Empower sequence labeling with task-aware neural language model,

L. Liu, J. Shang, X. Ren, F. F. Xu, H. Gui, J. Peng, and J. Han, “Empower sequence labeling with task-aware neural language model,” in AAAI, 2018

work page 2018
[3]

Neural architectures for named entity recognition,

G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer, “Neural architectures for named entity recognition,” in NAACL, 2016

work page 2016
[4]

Design challenges and misconceptions in named entity recognition,

L. Ratinov and R. Dan, “Design challenges and misconceptions in named entity recognition,” in CoNLL, 2009

work page 2009
[5]

Lexicon infused phrase embeddings for named entity resolution,

A. Passos, V . Kumar, and A. Mccallum, “Lexicon infused phrase embeddings for named entity resolution,” in CS, 2014

work page 2014
[6]

Deterministic dependency parsing of english text,

J. Nivre and M. Scholz, “Deterministic dependency parsing of english text,” in COLING. Association for Computational Linguistics, 2004, p. 64

work page 2004
[7]

Heterogeneous supervision for relation extraction: A representation learning approach,

L. Liu, X. Ren, Q. Zhu, S. Zhi, H. Gui, H. Ji, and J. Han, “Heterogeneous supervision for relation extraction: A representation learning approach,” in EMNLP, 2017

work page 2017
[8]

Supervised noun phrase coreference research: The ﬁrst ﬁfteen years,

V . Ng, “Supervised noun phrase coreference research: The ﬁrst ﬁfteen years,” in ACL, 2010

work page 2010

Show all 48 references

[9]

Statistical inference for probabilistic functions of ﬁnite state markov chains,

L. E. Baum and T. Petrie, “Statistical inference for probabilistic functions of ﬁnite state markov chains,” The annals of mathematical statistics, pp. 1554–1563, 1966. XXX JOURNAL, VOL. X, NO. X, XX XXXX 10 Sent1 The market opened sharply lower, with the Nikkei average down ne...

1966
[10]

Conditional random ﬁelds: Probabilistic models for segmenting and labeling sequence data,

J. Lafferty, A. McCallum, and F. C. Pereira, “Conditional random ﬁelds: Probabilistic models for segmenting and labeling sequence data,” in ICML, 2001

2001
[11]

Natural language processing (almost) from scratch,

R. Collobert, K. Kavukcuoglu, J. Weston, L. Bottou, P. Kuksa, and M. Karlen, “Natural language processing (almost) from scratch,”Journal of Machine Learning Research , vol. 12, no. 1, pp. 2493–2537, 2011

2011
[12]

Named entity recognition with bidirectional lstm-cnns,

J. P. Chiu and E. Nichols, “Named entity recognition with bidirectional lstm-cnns,” TACL, vol. 4, pp. 357–370, 2016

2016
[13]

Deep semantic role labeling with self-attention,

Z. Tan, M. Wang, J. Xie, Y . Chen, and X. Shi, “Deep semantic role labeling with self-attention,” in AAAI, 2018

2018
[14]

Long short-term memory-networks for machine reading,

J. Cheng, D. Li, and M. Lapata, “Long short-term memory-networks for machine reading,” in EMNLP, 2016

2016
[15]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in NIPS, 2017, pp. 5998–6008

2017
[16]

A structured self-attentive sentence embedding,

Z. Lin, M. Feng, C. N. d. Santos, M. Yu, B. Xiang, B. Zhou, and Y . Bengio, “A structured self-attentive sentence embedding,” in ICLR, 2017

2017
[17]

Self-attention with relative position representations,

P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” in NAACL, 2018

2018
[18]

A tutorial on hidden markov models and selected applications in speech recognition,

L. R. Rabiner, “A tutorial on hidden markov models and selected applications in speech recognition,” Proceedings of the IEEE , vol. 77, no. 2, pp. 257–286, 1989

1989
[19]

Early results for named entity recognition with conditional random ﬁelds, feature induction and web-enhanced lexicons,

A. Mccallum and W. Li, “Early results for named entity recognition with conditional random ﬁelds, feature induction and web-enhanced lexicons,” in CoNLL, 2003

2003
[20]

Use of support vector learning for chunk identiﬁcation,

T. Kudoh and Y . Matsumoto, “Use of support vector learning for chunk identiﬁcation,” in Fourth Conference on Computational Natural Lan- guage Learning and the Second Learning Language in Logic Workshop , 2000

2000
[21]

Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms,

M. Collins, “Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms,” in Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10 . Association for Computational Linguistics, 2002, pp. 1–8

2002
[22]

Bidirectional lstm-crf models for sequence tagging,

Z. Huang, X. Wei, and Y . Kai, “Bidirectional lstm-crf models for sequence tagging,” in Computer Science, 2015

2015
[23]

Learning tag dependencies for sequence tagging

Y . Zhang, H. Chen, Y . Zhao, Q. Liu, and D. Yin, “Learning tag dependencies for sequence tagging.” in IJCAI, 2018

2018
[24]

Transfer learning for sequence tagging with hierarchical recurrent networks,

Z. Yang, R. Salakhutdinov, and W. W. Cohen, “Transfer learning for sequence tagging with hierarchical recurrent networks,” in ICLR, 2017

2017
[25]

Segbot: A generic neural text segmentation model with pointer network

J. Li, A. Sun, and S. Joty, “Segbot: A generic neural text segmentation model with pointer network.” in IJCAI, 2018, pp. 4166–4172

2018
[26]

Disan: Direc- tional self-attention network for rnn/cnn-free language understanding,

T. Shen, T. Zhou, G. Long, J. Jiang, S. Pan, and C. Zhang, “Disan: Direc- tional self-attention network for rnn/cnn-free language understanding,” in AAAI, 2018

2018
[27]

Self- attentional acoustic models,

M. Sperber, J. Niehues, G. Neubig, S. St ¨uker, and A. Waibel, “Self- attentional acoustic models,” Interspeech, 2018

2018
[28]

Learning character-level representa- tions for part-of-speech tagging,

C. N. D. Santos and B. Zadrozny, “Learning character-level representa- tions for part-of-speech tagging,” in ICML, 2014

2014
[29]

Long short-term memory,

S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997

1997
[30]

Neural machine translation by jointly learning to align and translate,

D. Bahdanau, K. Cho, and Y . Bengio, “Neural machine translation by jointly learning to align and translate,” in ICLR, 2014

2014
[31]

Effective approaches to attention-based neural machine translation,

M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” in EMNLP, 2015

2015
[32]

Semi- supervised sequence tagging with bidirectional language models,

M. E. Peters, W. Ammar, C. Bhagavatula, and R. Power, “Semi- supervised sequence tagging with bidirectional language models,” in ACL, 2017

2017
[33]

Glove: Global vectors for word representation,

J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in EMNLP, 2014, pp. 1532–1543

2014
[34]

Understanding the difﬁculty of training deep feedforward neural networks,

X. Glorot and Y . Bengio, “Understanding the difﬁculty of training deep feedforward neural networks,” in Proceedings of the thirteenth international conference on artiﬁcial intelligence and statistics , 2010, pp. 249–256

2010
[35]

On the difﬁculty of training recurrent neural networks,

R. Pascanu, T. Mikolov, and Y . Bengio, “On the difﬁculty of training recurrent neural networks,” in ICML, 2013

2013
[36]

Over- ﬁtting in neural nets: Backpropagation, conjugate gradient, and early stopping,

C. Lee, G. Overﬁtting, R. Caruana, S. Lawrence, and L. Giles, “Over- ﬁtting in neural nets: Backpropagation, conjugate gradient, and early stopping,” in NIPS, 2000

2000
[37]

Semi-supervised multitask learning for sequence labeling,

M. Rei, “Semi-supervised multitask learning for sequence labeling,” in ACL, 2017

2017
[38]

Does higher order lstm have better accuracy for segmenting and labeling sequence data?

Y . Zhang, X. Sun, S. Ma, Y . Yang, and X. Ren, “Does higher order lstm have better accuracy for segmenting and labeling sequence data?” in COLING, 2017

2017
[39]

Deep contextualized word representations,

M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in NAACL, 2018

2018
[40]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in arXiv, 2018

2018
[41]

Contextual string embeddings for sequence labeling,

A. Akbik, D. Blythe, and R. V ollgraf, “Contextual string embeddings for sequence labeling,” in COLING, 2018, pp. 1638–1649

2018
[42]

Guided learning for bidirectional sequence classiﬁcation,

L. Shen, G. Satta, and A. Joshi, “Guided learning for bidirectional sequence classiﬁcation,” in ACL, 2007

2007
[43]

Structure regularization for structured prediction,

X. Sun, “Structure regularization for structured prediction,” in NIPS, 2014

2014
[44]

Semisupervised condensed nearest neighbor for part-of- speech tagging,

A. Søgaard, “Semisupervised condensed nearest neighbor for part-of- speech tagging,” in ACL, 2011

2011
[45]

Robust multilingual part-of- speech tagging via adversarial training,

M. Yasunaga, J. Kasai, and D. Radev, “Robust multilingual part-of- speech tagging via adversarial training,” in NAACL, 2017

2017
[46]

Feature-frequency–adaptive on-line training for fast and accurate natural language processing,

X. Sun, W. Li, H. Wang, and Q. Lu, “Feature-frequency–adaptive on-line training for fast and accurate natural language processing,” Computational Linguistics, vol. 40, no. 3, pp. 563–586, 2014

2014
[47]

A new recurrent neural crf for learning non-linear edge features,

S. Ma and X. Sun, “A new recurrent neural crf for learning non-linear edge features,” in arXiv preprint arXiv:1611.04233 , 2016

2016 arXiv
[48]

Neural models for sequence chunking,

F. Zhai, S. Potdar, B. Xiang, and B. Zhou, “Neural models for sequence chunking,” in Thirty-First AAAI Conference on Artiﬁcial Intelligence , 2017. XXX JOURNAL, VOL. X, NO. X, XX XXXX 11 PLACE PHOTO HERE Wei Wei received the PhD degree from Huazhong University of Science and T...

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

End-to-end sequence labeling via bi-directional lstm-cnns-crf,

X. Ma and E. Hovy, “End-to-end sequence labeling via bi-directional lstm-cnns-crf,” in ACL, 2016

work page 2016

[2] [2]

Empower sequence labeling with task-aware neural language model,

L. Liu, J. Shang, X. Ren, F. F. Xu, H. Gui, J. Peng, and J. Han, “Empower sequence labeling with task-aware neural language model,” in AAAI, 2018

work page 2018

[3] [3]

Neural architectures for named entity recognition,

G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer, “Neural architectures for named entity recognition,” in NAACL, 2016

work page 2016

[4] [4]

Design challenges and misconceptions in named entity recognition,

L. Ratinov and R. Dan, “Design challenges and misconceptions in named entity recognition,” in CoNLL, 2009

work page 2009

[5] [5]

Lexicon infused phrase embeddings for named entity resolution,

A. Passos, V . Kumar, and A. Mccallum, “Lexicon infused phrase embeddings for named entity resolution,” in CS, 2014

work page 2014

[6] [6]

Deterministic dependency parsing of english text,

J. Nivre and M. Scholz, “Deterministic dependency parsing of english text,” in COLING. Association for Computational Linguistics, 2004, p. 64

work page 2004

[7] [7]

Heterogeneous supervision for relation extraction: A representation learning approach,

L. Liu, X. Ren, Q. Zhu, S. Zhi, H. Gui, H. Ji, and J. Han, “Heterogeneous supervision for relation extraction: A representation learning approach,” in EMNLP, 2017

work page 2017

[8] [8]

Supervised noun phrase coreference research: The ﬁrst ﬁfteen years,

V . Ng, “Supervised noun phrase coreference research: The ﬁrst ﬁfteen years,” in ACL, 2010

work page 2010