Pith. sign in

REVIEW 4 major objections 5 minor 59 references

Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding a lexical-relation classification task to BERT pretraining yields a model that beats standard BERT on 9 of 10 GLUE tasks and improves lexical simplification by up to 8.2 points.

desk verdict A promising idea undermined by a missing control: LIBERT's gains over BERT may be due to extra optimizer steps and data, not lexical knowledge. read the letter →

arxiv 1909.02339 v2 pith:PQO4RAFK submitted 2019-09-05 cs.CL

Anne Lauscher , Ivan Vulić , Edoardo Maria Ponti , Anna Korhonen , Goran Glavaš This is my paper · ORCID
classification cs.CL
keywords BERTmulti-taskpretraininglexicalrelationclassificationsemanticsimilaritysimplificationGLUEbenchmarkWordNetdistributionalsemantics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Unsupervised language models such as BERT learn word meaning from text co-occurrence alone, which makes it hard for them to separate genuine similarity (car/automobile) from broader topical relatedness (car/road). This paper asks whether that gap can be closed by adding an explicit pretraining task that classifies word pairs as semantically similar or not, using synonym and hypernymy pairs from WordNet and Roget's Thesaurus. The authors report that the resulting model, LIBERT, beats a same-configuration BERT on 9 of 10 GLUE test tasks after 2M pretraining steps (and ties on QNLI), with the largest gains on linguistic acceptability and the diagnostic AX set, and improves lexical simplification on all three evaluated benchmarks, up to 8.2 accuracy points. If true, the result matters because it offers a low-cost route to inject curated lexical knowledge into large pretrained encoders, and because it shows that distributional pretraining leaves a measurable, correctable deficit in word-level similarity.

What carries the argument

The load-bearing component is the Lexical Relation Classification (LRC) objective: a two-way softmax classifier on the [CLS] representation of a BERT-compatible word-pair input that decides whether the two words stand in a true semantic-similarity relation. Positive examples are synonym and direct hypernym/hyponym pairs from WordNet and Roget; negative examples are built by taking, for each positive pair, the words from the same mini-batch that are closest in a fastText space, so the classifier must learn to separate genuine similarity from near-miss relatedness. LRC's gradient is applied to the transformer in an alternating regime with MLM and NSP batches, which is the mechanism by which lexical knowledge is written into the contextualized representations.

What would settle it

Train LIBERT with the same architecture and data but randomize the similarity labels of the constraint pairs; if its GLUE and lexical-simplification scores still beat BERT, the reported gains are not caused by lexical content. A complementary check is to hold out a random subset of WordNet synonym pairs from pretraining and test the LRC head on them: near-chance accuracy on held-out pairs would indicate the classifier did not learn a transferable notion of similarity.

Watch

Extended reading notes

Core claim

The paper's central claim is that a binary lexical-relation classification head, trained alongside BERT's masked language modeling and next sentence prediction on 1.35M synonym and direct hypernym/hyponym pairs, steers contextualized representations toward true word-level semantic similarity. With all other hyperparameters and pretraining data kept identical, the resulting LIBERT model outperforms standard BERT on 9 of 10 GLUE test tasks after 2M steps and matches it on QNLI; the biggest margins appear on CoLA, AX, and SST-2, and the advantage persists when pretraining is extended from 1M to 2M steps. On lexical simplification, LIBERT improves both candidate generation and the full replacement pipeline on all three datasets, with an accuracy gain of up to 8.2 points on LexMTurk. The authors read these results as evidence that clean external lexical knowledge complements, rather than duplicates, the distributional signal in large-scale language modeling.

Load-bearing premise

The result depends on the LRC task teaching a generalizable sense of similarity rather than memorizing the 1.35M training pairs or their lexical form.

Editorial extensions

If this is right

  • On the GLUE test sets after 2M pretraining steps, LIBERT beats BERT on 9 of 10 tasks and matches it on QNLI; the largest deltas are on CoLA (+6.5), AX (+2.5), and SST-2 (+1.1).
  • On the GLUE diagnostic set, LIBERT's largest improvements are in Lexical Semantics (up to +29.7 at 2M), with large gains also in Logic and Knowledge & Common Sense at 1M, suggesting lexical knowledge partially supports factual and logical inference.
  • In lexical simplification, LIBERT improves both candidate generation (F1 gains of +1.3 to +3.0 points at 2M across datasets) and full-pipeline accuracy, with the largest gain +8.2 points on LexMTurk.
  • The advantage of lexical injection does not disappear as pretraining continues: deltas over BERT after 2M MLM+NSP steps are comparable to or larger than after 1M.
  • Because LIBERT and BERT are trained from scratch under identical settings, the reported differences isolate the effect of the LRC task rather than other pretraining variables.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same auxiliary-task recipe could be applied to other lexical relations (antonymy, meronymy, lexical entailment) and to other pretrained encoders; the paper tests only synonymy and hypernymy in English, so this is an extension, not a result.
  • The diagnostic gains in Factivity and Morphological Negation suggest the injected constraints sharpen the model's compositionality, so a targeted test would be to measure LIBERT on morphological generalization or lexical entailment datasets rather than broad GLUE tasks.
  • If the LRC signal is truly complementary, training both models much longer (or on more text) should preserve at least part of the gap; a direct test would extend their pretraining beyond 2M steps and track deltas.
  • The negative-sampling choice is a likely active ingredient: replacing fastText-based near-miss negatives with random non-synonym pairs might shrink or erase the gains, which would indicate that contrastive difficulty, not just pair labels, drives the specialization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LIBERT, a variant of BERT that adds a third pretraining objective, Lexical Relation Classification (LRC), to the standard masked language modeling and next sentence prediction tasks. The LRC objective is trained on roughly 1.35M synonym and direct hyponym-hypernym pairs from WordNet and Roget's Thesaurus, with negatives constructed from in-batch nearest neighbors in a fastText space. The authors train LIBERT and a vanilla BERT baseline from scratch under the same configuration, evaluate on the GLUE benchmark and on three lexical simplification datasets, and report that LIBERT outperforms BERT on 9/10 GLUE test tasks (and ties on QNLI) after 2M reported MLM+NSP steps, with the largest margins on CoLA, AX, and SST-2, and that it improves lexical simplification on all three benchmarks with gains up to 8.2 points. They also present a fine-grained analysis on the GLUE Diagnostic dataset showing larger gains in categories such as Lexical Semantics, Logic, and Knowledge & Common Sense.

Significance. If the reported gains are causally attributable to the injected lexical knowledge, the paper would make a useful contribution: it demonstrates a simple, general way to complement distributional pretraining with structured lexical resources and shows downstream benefits beyond tasks where word-level similarity is an obvious ingredient. Strengths of the work include the design decision to train both models from scratch with the same hyperparameters, the use of external benchmarks that are not used to fit the model, the diagnostic breakdown on GLUE's Diagnostic set, and the fact that gains persist after 2M steps. The lexical simplification experiments are also a sensible similarity-focused testbed. However, the central causal claim is not yet established because the comparison is not matched in total optimization budget, and the reported differences on several GLUE tasks are within typical run-to-run variance for these benchmarks. The core idea is promising and the evidence is suggestive, but the current manuscript needs a stronger experimental design or a more modest interpretation before the attribution claim can be accepted.

major comments (4)
  1. [Section 4.1, Training and Evaluation] The experimental comparison is confounded by an unmatched training budget. The paper states that LIBERT updates parameters in an alternating regime: one batch of MLM+NSP followed by one batch of LRC. Therefore, after the reported '2M MLM+NSP steps,' LIBERT has additionally consumed 2M LRC batches, each containing k positive and 2k negative word pairs, while vanilla BERT has received only the 2M sentence batches. The two models thus differ not only in the presence of lexical constraints but also in total optimizer steps, total data volume, and total gradient signal. The central claim that lexico-semantic knowledge causes the observed gains therefore requires additional controls, such as (a) a BERT baseline trained for the same total number of updates with an auxiliary task on shuffled or non-lexical constraint pairs, or (b) a BERT baseline trained for the same total number of updates with an auxiliary task of comparable difficulty that contains no semantic-similarity information. Without such a control, the reported gains could plausibly reflect additional compute and data rather than semantic specialization.
  2. [Section 4.2, Table 2] No variance or significance information is reported, and several differences are small enough to be attributable to random seed variation. For example, at 2M steps the test-set differences are +0.2 on RTE, +0.3 on QQP F1, +0.7 on MNLI-mm, and +0.0 on QNLI; single-run differences of this magnitude are typically within the run-to-run noise for GLUE tasks. The paper should report means and standard deviations over multiple random seeds (at least for fine-tuning, and preferably also for pretraining or with several pretraining seeds), and indicate whether the differences are statistically significant. This issue is load-bearing because the headline claim is an improvement on '9 out of 10' tasks, many of which rest on small single-run margins.
  3. [Section 4.2, Table 3] The fine-grained Diagnostic analysis reports very large relative gains on small subsets (e.g., Factivity +281.7% at 1M steps), but no information is given about the number of examples in each fine-grained category or the variance of the R3 coefficient. Large percentage changes on small subsets should be interpreted cautiously. This is not the central claim of the paper, but the analysis as presented may overstate the reliability of the diagnostic conclusions.
  4. [Section 5, Results and Discussion] The lexical simplification gains are also reported as single-run numbers without variance or significance testing. In Table 4, the differences in accuracy are sometimes substantial (e.g., +8.2 on LexMTurk at 1M steps), but the precision/recall/F1 differences in candidate generation are often small (e.g., +0.0098 recall on LexMTurk at 1M steps). The claim that LIBERT 'consistently outperforms BERT' would be strengthened by multiple seeds and a statement of significance, especially because the pipeline includes additional features such as fastText similarity and word frequency that are held fixed across models.
minor comments (5)
  1. [Section 4.1, Training and Evaluation] There is a typo 'the the original work' in the footnote about batch size. The footnote itself is useful context, but the sentence should be corrected.
  2. [Table 4] The column header 'accurracy' is misspelled; it should be 'accuracy'.
  3. [Section 3.2, Eq. (1)] The notation WLRC∈RH×2 and bLRC∈R2 is fine, but it would be clearer to state explicitly that the LRC classifier is used only during pretraining and is discarded or re-initialized during downstream fine-tuning. The text implies this, but an explicit statement would avoid ambiguity.
  4. [Section 5, Experimental Setup] The description of the BERT-LS approach is clear, but the paper should specify whether the LRC classifier is used at all during lexical simplification inference, since only the MLM predictor appears to be used. Clarifying this would help readers understand exactly which components differ between BERT and LIBERT at evaluation time.
  5. [References] The two 'ERNIE' references (Zhang et al., 2019a and 2019b) appear to have very similar titles and one appears twice with different arXiv numbers; the authors should check whether these are meant to be distinct papers and cite them appropriately.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the auxiliary LRC objective is trained on external lexicons and all claimed gains are measured on held-out external benchmarks.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs. LIBERT's only addition to BERT is the lexical relation classification loss L_LRC (Eqs. 1-2), trained on synonym/hypernym pairs from WordNet and Roget's Thesaurus; the negative examples are generated from fastText similarities among those same constraint words. None of these constraints or losses are fitted to GLUE or to the lexical simplification test sets. The reported outcomes are computed by standard fine-tuning on each downstream task's training split and evaluation on held-out test sets, so the scores are not defined in terms of the training objective. The self-citations to Glavaš and Vulić (2018), Vulić et al. (2018), Ponti et al. (2018, 2019), and Mrkšić et al. (2017) are prior specialization methods for static embeddings with their own external evaluations; they motivate the design choice but do not function as a uniqueness theorem or as an ansatz that forces the reported GLUE/LS numbers. There is a genuine experimental confound, noted separately from circularity: the alternating training regime gives LIBERT one extra LRC gradient update per MLM+NSP step, so LIBERT and BERT differ in total updates and data volume as well as in lexical signal; this weakens causal attribution but does not make any prediction equal to an input by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality and transferability of external lexical constraints and on the design of the LRC task. No numerical parameter is fitted to the downstream target results; the architecture, learning rates, and batch sizes are taken from BERT (Devlin et al., 2019) or fixed equivalently for both models.

assumptions (4)
  • domain assumption WordNet synonymy and direct hypernymy are valid operationalizations of true semantic similarity.
    Used to build the positive constraint set in Section 3.2; if these relations do not match the notion of similarity relevant to downstream tasks, the auxiliary task teaches a mismatched signal.
  • domain assumption The [CLS] vector of a word-pair input encodes sufficient relation information for a linear classifier to learn the lexical relation.
    Equation (1) applies softmax to x_CLS; the paper assumes this representation is informative and that this learning transfers to contextualized word representations.
  • domain assumption Alternating optimization of BERT's MLM+NSP objective and the LRC objective preserves language modeling quality while adding lexical signal.
    Section 4.1 describes the balanced alternating regime; if the LRC gradients interfere destructively, the comparison would be confounded.
  • domain assumption fastText nearest neighbors in the same mini-batch provide useful hard negatives for the relation classifier.
    Negative sampling in Section 3.2 depends on Xaux; the difficulty of the negatives determines whether the classifier learns to distinguish similarity from relatedness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity." pith.science (2026). https://pith.science/paper/PQO4RAFK

@misc{pith2026190902339,
  author       = {Pith},
  title        = {Pith review of: Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PQO4RAFK}},
  note         = {Machine review of arXiv:1909.02339}
}
read the original abstract

Unsupervised pretraining models have been shown to facilitate a wide range of downstream NLP applications. These models, however, retain some of the limitations of traditional static word embeddings. In particular, they encode only the distributional knowledge available in raw text corpora, incorporated through language modeling objectives. In this work, we complement such distributional knowledge with external lexical knowledge, that is, we integrate the discrete knowledge on word-level semantic similarity into pretraining. To this end, we generalize the standard BERT model to a multi-task learning setting where we couple BERT's masked language modeling and next sentence prediction objectives with an auxiliary task of binary word relation classification. Our experiments suggest that our "Lexically Informed" BERT (LIBERT), specialized for the word-level semantic similarity, yields better performance than the lexically blind "vanilla" BERT on several language understanding tasks. Concretely, LIBERT outperforms BERT in 9 out of 10 tasks of the GLUE benchmark and is on a par with BERT in the remaining one. Moreover, we show consistent gains on 3 benchmarks for lexical simplification, a task where knowledge about word-level semantic similarity is paramount.

Figures

Figures reproduced from arXiv: 1909.02339 by the authors.

Figure 1
Figure 1. Architecture of LIBERT – lexically-informed [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Accuracy over time for BERT and LIBERT on (a) SST-2 and (b) MRPC on the corresponding dev sets. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 44 canonical work pages

  1. [1]

    Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. https://tac.nist.gov//publications/2009/additional.papers/RTE5_overview.proceedings.pdf The Fifth PASCAL recognizing textual entailment challenge. In Proceedings of TAC

    work page 2009

  2. [2]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. http://arxiv.org/abs/1607.04606 Enriching word vectors with subword information . Transactions of the ACL, 5:135--146

    arXiv 2017

  3. [3]

    Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. https://doi.org/10.18653/v1/S17-2001 S em E val-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation . In Proceedings of S em E val , pages 1--14

    doi:10.18653/v1/s17-2001 2017

  4. [4]

    Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. 2018. http://static.hongbozhang.me/doc/STAT_441_Report.pdf Quora question pairs . Technical report, University of Waterloo

    work page 2018

  5. [5]

    Jan De Belder and Marie-Francine Moens. 2010. https://lirias.kuleuven.be/retrieve/120012 Text simplification for children . In Proceedings of the SIGIR workshop on accessible search systems, pages 19--26

    work page 2010

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://www.aclweb.org/anthology/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of NAACL-HLT, pages 4171--4186

    work page 2019

  7. [7]

    Dolan and Chris Brockett

    William B. Dolan and Chris Brockett. 2005. https://www.aclweb.org/anthology/I05-5002 Automatically constructing a corpus of sentential paraphrases . In Proceedings of the Third International Workshop on Paraphrasing (IWP2005)

    work page 2005

  8. [8]

    Manaal Faruqui. 2016. http://www.manaalfaruqui.com/papers/thesis.pdf Diverse Context for Learning Word Representations . Ph.D. thesis, Carnegie Mellon University

    work page 2016

Show all 59 references
  1. [9]

    Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. http://www.aclweb.org/anthology/N15-1184 Retrofitting word vectors to semantic lexicons . In Proceedings of NAACL-HLT, pages 1606--1615

    2015

  2. [10]

    Goran Glava s and Sanja S tajner. 2015. https://doi.org/10.3115/v1/P15-2011 Simplifying lexical simplification: Do we need simplified corpora? In Proceedings of ACL-IJCNLP, pages 63--68, Beijing, China

    2015 doi

  3. [11]

    Goran Glava s and Ivan Vuli \'c . 2018. https://www.aclweb.org/anthology/P18-1004 Explicit retrofitting of distributional word vectors . In Proceedings of ACL, pages 34--45

    2018

  4. [12]

    Felix Hill, Roi Reichart, and Anna Korhonen. 2015. https://doi.org/10.1162/COLI\_a\_00237 Simlex-999: E valuating semantic models with (genuine) similarity estimation . Computational Linguistics, 41(4):665--695

    2015 doi

  5. [13]

    Colby Horn, Cathryn Manduca, and David Kauchak. 2014. https://doi.org/10.3115/v1/P14-2075 Learning a lexical simplifier using W ikipedia . In Proceedings of ACL, pages 458--463

    2014 doi

  6. [14]

    Hwiyeol Jo and Stanley Jungkyu Choi. 2018. http://arxiv.org/abs/1804.07946 Extrofitting: E nriching word representation and its vector space with semantic lexicons . CoRR, abs/1804.07946

    2018 arXiv

  7. [15]

    Aishwarya Kamath, Jonas Pfeiffer, Edoardo Maria Ponti, Goran Glava s , and Ivan Vuli \'c . 2019. https://www.aclweb.org/anthology/W19-4310 Specializing distributional vectors of all words for lexical entailment . In Proceedings of the 4th Workshop on Representation Learning fo...

    2019

  8. [16]

    Douwe Kiela, Felix Hill, and Stephen Clark. 2015. http://aclweb.org/anthology/D15-1242 Specializing word embeddings for similarity or relatedness . In Proceedings of EMNLP, pages 2044--2048

    2015

  9. [17]

    Joo-Kyung Kim, Gokhan Tur, Asli Celikyilmaz, Bin Cao, and Ye-Yi Wang. 2016. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/09/intent-detection-semantically.pdf Intent detection using semantically enriched word embeddings . In Proceedings of SLT

    2016

  10. [18]

    Barbara Ann Kipfer. 2009. http://www.randomhouse.com/highschool/catalog/display.pperl?isbn=9780440242697 Roget's 21st Century Thesaurus (3rd Edition) . Philip Lief Group

    2009

  11. [19]

    Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, and Yu Hu. 2015. http://www.aclweb.org/anthology/P15-1145 Learning semantic word embeddings based on ordinal knowledge constraints . In Proceedings of ACL, pages 1501--1511

    2015

  12. [20]

    Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2019 a . https://arxiv.org/pdf/1909.07606.pdf K-BERT : Enabling language representation with knowledge graph . arXiv preprint arXiv:1909.07606

    2019 arXiv

  13. [21]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 b . https://arxiv.org/pdf/1907.11692.pdf Ro BERT a: A robustly optimized bert pretraining approach . arXiv preprint arXiv:1907.11692

    2019 arXiv

  14. [22]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S Corrado, and Jeffrey Dean. 2013. https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf Distributed representations of words and phrases and their compositionality . I...

    2013

  15. [23]

    George A. Miller. 1995. https://doi.org/10.1145/219717.219748 WordNet : A lexical database for English . Commun. ACM, 38(11):39--41

    1995

  16. [24]

    Nikola Mrk s i\' c , Diarmuid \' O S \' e aghdha, Blaise Thomson, Milica Ga s i\' c , Lina Maria Rojas - Barahona, Pei - Hao Su, David Vandyke, Tsung - Hsien Wen, and Steve Young. 2016. http://aclweb.org/anthology/N/N16/N16-1018.pdf Counter-fitting word vectors to linguistic c...

    2016

  17. [25]

    Nikola Mrk s i\'c, Ivan Vuli\' c , Diarmuid \'O S\'eaghdha , Ira Leviant, Roi Reichart, Milica Ga s i\' c , Anna Korhonen, and Steve Young. 2017. http://aclweb.org/anthology/Q/Q17/Q17-1022.pdf Semantic specialisation of distributional word vector spaces using monolingual and c...

    2017

  18. [26]

    Roberto Navigli and Simone Paolo Ponzetto. 2012. https://core.ac.uk/download/pdf/82665941.pdf BabelNet: T he automatic construction, evaluation and application of a wide-coverage multilingual semantic network . Artificial Intelligence, 193:217--250

    2012

  19. [27]

    Kim Anh Nguyen, Maximilian K\" o per, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017. https://www.aclweb.org/anthology/D17-1022 Hierarchical embeddings for hypernymy detection and directionality . In Proceedings of EMNLP, pages 233--243

    2017

  20. [28]

    Dominique Osborne, Shashi Narayan, and Shay Cohen. 2016. https://arxiv.org/abs/1509.01007 Encoding prior knowledge with eigenword embeddings . Transactions of the ACL, 4:417--430

    2016 arXiv

  21. [29]

    Gustavo Paetzold and Lucia Specia. 2016. https://www.aclweb.org/anthology/L16-1491 Benchmarking lexical simplification systems . In Proceedings of LREC , pages 3074--3080, Portoro z , Slovenia

    2016

  22. [30]

    Gustavo H Paetzold and Lucia Specia. 2017. https://www.jair.org/index.php/jair/article/download/11091/26278/ A survey on lexical simplification . Journal of Artificial Intelligence Research, 60:549--593

    2017

  23. [31]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of NAACL-HLT, pages 2227--2237

    2018 doi

  24. [32]

    Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith

    Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. https://www.aclweb.org/anthology/D19-1005.pdf Knowledge enhanced contextual word representations . In Proceedings of the EMNLP-IJCNLP, pages 43--54

    2019

  25. [33]

    Fabio Petroni, Tim Rockt \"a schel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. https://www.aclweb.org/anthology/D19-1250.pdf Language models as knowledge bases? In Proceedings of EMNLP-IJCNLP, pages 2463--2473

    2019

  26. [34]

    Edoardo Maria Ponti, Ivan Vuli \'c , Goran Glava s , Nikola Mrk s i \'c , and Anna Korhonen. 2018. https://www.aclweb.org/anthology/D18-1026 Adversarial propagation and zero-shot cross-lingual transfer of word vector specialization . In Proceedings of EMNLP, pages 282--293

    2018

  27. [35]

    Edoardo Maria Ponti, Ivan Vuli \'c , Goran Glava s , Roi Reichart, and Anna Korhonen. 2019. https://www.aclweb.org/anthology/D19-1226.pdf Cross-lingual semantic specialization via lexical relation induction . In Proceedings of the EMNLP-IJCNLP, pages 2206--2217

    2019

  28. [36]

    Jipeng Qiang, Yun Li, Zhu Yi, Yunhao Yuan, and Xindong Wu. 2019. https://arxiv.org/pdf/1907.06226.pdf A simple BERT -based approach for lexical simplification . arXiv preprint arXiv:1907.06226

    2019 arXiv

  29. [37]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. https://www.cs.ubc.ca/ amuham01/LING530/papers/radford2018improving.pdf Improving language understanding by generative pre-training . OpenAI Technical Report

    2018

  30. [38]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://www.techbooky.com/wp-content/uploads/2019/02/Better-Language-Models-and-Their-Implications.pdf Language models are unsupervised multitask learners . OpenAI Blog, 1:8

    2019

  31. [39]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of EMNLP, pages 2383--2392

    2016 doi

  32. [40]

    Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. https://www.aclweb.org/anthology/D18-1299 Towards universal dialogue state tracking . In Proceedings of EMNLP, pages 2780--2786

    2018

  33. [41]

    Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015. http://www.aclweb.org/anthology/K15-1026 Symmetric pattern based word embeddings for improved word similarity prediction . In Proceedings of CoNLL, pages 258--267

    2015

  34. [42]

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. https://www.aclweb.org/anthology/D13-1170 Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of EMNLP, pages 1631--1642

    2013

  35. [43]

    Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2019. https://arxiv.org/pdf/1907.12412.pdf ERNIE 2.0: A continual pre-training framework for language understanding . arXiv preprint arXiv:1907.12412

    2019 arXiv

  36. [44]

    Wilson L. Taylor. 1953. https://doi.org/10.1177/107769905303000401 `` Cloze procedure'': A new tool for measuring readability . Journalism Bulletin, 30(4):415--433

    1953 doi

  37. [45]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention is all you need . In Proceedings of NeurIPS, pages 5998--6008

    2017

  38. [46]

    Ivan Vuli \'c . 2018. https://www.aclweb.org/anthology/W18-3018 Injecting lexical contrast into word vectors by guiding vector space specialisation . In Proceedings of the 3rd Workshop on Representation Learning for NLP , pages 137--143

    2018

  39. [47]

    Ivan Vuli\' c , Goran Glava s , Nikola Mrk s i\' c , and Anna Korhonen. 2018. https://arxiv.org/pdf/1805.03228.pdf Post-specialisation: Retrofitting vectors of words unseen in lexical resources . In Proceedings of NAACL-HLT, pages 516--527

    2018 arXiv

  40. [48]

    Ivan Vuli \'c and Nikola Mrk s i \'c . 2018. https://www.aclweb.org/anthology/N18-1103 Specialising word vectors for lexical entailment . In Proceedings of NAACL-HLT, pages 1134--1145

    2018

  41. [49]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://www.aclweb.org/anthology/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the Blacbox NLP Workshop, pages 353--355

    2018

  42. [50]

    Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. https://www.nyu.edu/projects/bowman/neural_network_acceptability.pdf Neural network acceptability judgments . Transactions of the ACL, 7:625--641

    2019

  43. [51]

    John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. http://aclweb.org/anthology/Q/Q15/Q15-1025.pdf From paraphrase database to compositional paraphrase model and back . Transactions of the ACL , 3:345--358

    2015

  44. [52]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://www.aclweb.org/anthology/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of NAACL-HLT, pages 1112--1122

    2018

  45. [53]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. https://pdfs.semanticscholar.org/c685/0869aa5e78a107c378d2e8bfa39633158c0c.pdf Google's neural machine translation system: Bri...

    2016 arXiv

  46. [54]

    Mo Yu and Mark Dredze. 2014. http://www.aclweb.org/anthology/P14-2089 Improving lexical embeddings with semantic knowledge . In Proceedings of ACL, pages 545--550

    2014

  47. [55]

    Jingwei Zhang, Jeremy Salwen, Michael Glass, and Alfio Gliozzo. 2014. http://www.aclweb.org/anthology/D14-1161 Word semantic representations using Bayesian probabilistic tensor factorization . In Proceedings of EMNLP, pages 1522--1531

    2014

  48. [56]

    Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019 a . https://www.aclweb.org/anthology/P19-1139 ERNIE : E nhanced language representation with informative entities . In Proceedings of ACL, pages 1441--1451

    2019

  49. [57]

    Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019 b . https://arxiv.org/pdf/1905.07129.pdf ERNIE : Enhanced language representation with informative entities . arXiv preprint arXiv:1905.07129

    2019 arXiv

  50. [58]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.