Pith. sign in

REVIEW 4 major objections 4 minor 41 references

SenseBERT: Driving Some Sense into BERT

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pre-training a language model to predict word supersenses improves lexical understanding.

desk verdict Good idea, weakly supported: the sense-prediction loss is novel but never isolated from vocabulary, masking, and input-embedding changes, and the loss itself is mathematically biased toward uniform predictions over allowed senses. read the letter →

arxiv 1908.05646 v2 pith:A3ELOQPD submitted 2019-08-15 cs.CL cs.LG

Yoav Levine , Barak Lenz , Or Dagan , Ori Ram , Dan Padnos , Or Sharir , Shai Shalev-Shwartz , Amnon Shashua
show 1 more author
Yoav Shoham
This is my paper · ORCID
classification cs.CLcs.LG
keywords wordsensedisambiguationsupersensetaggingBERTweaksupervisionNetlexicalsemanticsinContextlanguagemodelpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SenseBERT claims that a language model can be taught word meanings during pre-training without any human sense annotations by adding a second prediction task: alongside guessing a masked word's identity, the model must guess its WordNet supersense, meaning its coarse semantic class such as noun.food or verb.creation. The training signal is weak, because every supersense WordNet allows for the masked word is treated as a possible label, but the paper argues that across a large corpus the correct sense is consistently reinforced while the competing labels cancel out. The result is that the model's representations carry more extractable lexical semantics: a frozen SenseBERT outperforms much larger plain BERT models on supersense disambiguation, and the largest version sets a then-state-of-the-art score on the Word in Context task, which asks whether a word is used in the same sense in two different contexts. This matters because it shows a path to injecting semantic knowledge into self-supervised language models using only an existing lexical database.

What carries the argument

The load-bearing mechanism is a parallel external mapping $S \in \mathbb{R}^{d \times 45}$ from the transformer's hidden states to the 45 WordNet supersense categories, trained jointly with BERT's word mapping $W$. For each masked word, the loss combines an allowed-senses term $-\log \sum_{s \in \mathcal{A}(w)} p(s \mid \text{context})$ with a regularization term that keeps the predicted distribution close to uniform over the allowed senses, and the same matrix $S$ is also added to the input embeddings through a fixed word-to-supersenses 0/1 matrix $M$, so the input vector becomes $v_{\text{input}} = (W + SM)x + p$. The supersense signal thus shapes the representations the model sees as well as the predictions it is scored on.

What would settle it

Train SenseBERT on a corpus from which one common sense of a polysemous word such as 'bass' as a fish has been removed, then evaluate on WiC and SemEval-SS items that require that sense; if accuracy on those items collapses while other-sense items stay high, the soft-label cancellation mechanism is not doing what the paper claims.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that adding a weakly supervised word-supersense prediction objective to BERT's masked-language-model pre-training produces a lexical-semantic level language model: the same transformer that predicts word forms also learns to predict meaning categories. Concretely, SenseBERT BASE scores 75.6 frozen and 83.0 fine-tuned on the SemEval-SS supersense disambiguation benchmark, versus 65.1 and 79.2 for BERT BASE, while SenseBERT LARGE reaches 72.14 on the Word in Context task, surpassing BERT LARGE at 69.6 and also prior WordNet-infused models. The paper interprets these gains as evidence that word-form-level self-supervision alone leaves a measurable gap in lexical semantics, and that a sense-level auxiliary task can largely close that gap without human annotation.

Load-bearing premise

The method assumes that, over a large unlabeled corpus, the correct supersense of a masked word is reinforced by its context more often than the other supersenses WordNet allows, so the noise from wrong soft labels averages out; if corpus sense frequencies are skewed, the model could systematically strengthen incorrect supersenses.

Editorial extensions

If this is right

  • Supersense disambiguation on SemEval-SS improves by more than 10 points in the frozen-embedding setting and by about 4 points after fine-tuning for both BASE and LARGE model sizes.
  • A single SenseBERT LARGE model achieves 72.14 on the Word in Context task, which at publication time was the state of the art and 2.5 points above BERT LARGE.
  • Lexical-semantic gains do not come at the cost of general understanding: SenseBERT BASE scores 77.9 on GLUE, on par with a same-data BERT BASE at 77.5.
  • The learned supersense matrix organizes the 45 supersenses into meaningful clusters, giving the model an explicit sense-level view of raw text during pre-training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the soft-label cancellation assumption could be stress-tested by constructing corpora with artificially skewed sense frequencies and measuring whether rare senses stop being learned.
  • Beyond the paper: because the supersense signal is word-level and does not depend on the surface form, the same recipe could be applied to other lexical inventories or to languages that lack sense-annotated corpora.
  • Beyond the paper: the enlarged 60K vocabulary adds 23 to 30 million parameters over vanilla BERT, so a controlled comparison using a same-size vocabulary without the sense head would isolate how much of the WiC gain comes from sense supervision rather than added capacity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces SenseBERT, a BERT-style model pretrained with an auxiliary masked-word WordNet supersense prediction objective. The model is trained to predict both the masked word and its supersenses, using a soft-labeling scheme over WordNet's allowed supersenses, and it augments input embeddings with static supersense vectors. The authors also enlarge the vocabulary to 60K tokens and bias the masking strategy toward single-supersense words. They evaluate on SemEval-SS, a supersense version of the SemEval WSD benchmark, and on WiC from SuperGLUE, reporting that SenseBERT BASE outperforms BERTBASE and BERTLARGE on SemEval-SS and that SenseBERT LARGE achieves 72.14 on WiC, a state-of-the-art score at the time. They also report GLUE results showing no overall degradation relative to a BERTBASE baseline trained on the same pretraining data.

Significance. If the reported gains are attributable to the sense-level pretraining objective, this is a meaningful contribution: it demonstrates that weak lexical supervision from WordNet can be injected at scale without human annotation, and the frozen-embedding probe improvements in Table 2 are particularly striking. The authors also deserve credit for training a BERTBASE baseline on the same pretraining data for GLUE, which controls for corpus differences. However, the paper does not isolate the proposed auxiliary loss from other architectural and training changes, and both main evaluation tasks share the same supersense label space used in pretraining. The significance of the central claim is therefore conditional on additional evidence that the gains come from the sense-prediction objective itself.

major comments (4)
  1. [Section 3.2-3.5; Tables 2-3] No ablation isolates the auxiliary supersense loss. SenseBERT BASE differs from BERTBASE in at least three ways: the 60K-token vocabulary versus BERT's 30K (Section 3.4), the single-supersense-biased masking strategy (Section 3.5), and the sense-augmented input embeddings of Eq. 8 (Section 3.3). The reported comparisons compare only the full combination against vanilla BERT; an arm that keeps the same vocabulary, masking, and input embeddings but removes the LSLM loss (Eq. 7) is necessary to attribute the SemEval-SS and WiC gains to the sense-prediction objective. Without such an ablation, the central claim is underdetermined.
  2. [Section 3.2, Eqs. (4)-(7)] The combined loss has a context-independent minimizer: for a fixed masked word w, LSLM is minimized by the uniform distribution over the allowed supersenses A(w), because L_allowed is maximized when the total probability on A(w) is 1, while L_reg penalizes any non-uniformity within A(w). Context sensitivity can only enter indirectly, through averaging over different masked words that share a context. The paper should address this property explicitly or provide evidence that the model's supersense predictions are context-sensitive as a direct result of LSLM rather than of the word-form LM or the input embeddings.
  3. [Sections 5.2-5.3; Tables 2-4] No error bars, confidence intervals, or multiple-seed results are reported, so it is unclear whether differences of 1-3 points (e.g., 70.3 versus 69.6 on WiC, 77.9 versus 77.5 on GLUE) are significant. The claim of 'significantly improved' lexical understanding needs variance estimates or at least repeated fine-tuning runs. In addition, the BERTLARGE WiC baseline is taken from the published SuperGLUE score rather than re-evaluated under the same fine-tuning protocol as SenseBERT, which weakens the comparability of the WiC comparison.
  4. [Section 5.3; Table 3] WiC's label scheme is deliberately defined over WordNet supersenses, the same 45-category inventory that SenseBERT is pretrained to predict. This alignment can inflate the apparent improvement: the model is trained to output exactly the label space used to define positive and negative WiC pairs. Evaluating on a task whose labels do not coincide with the pretraining taxonomy, or performing an analysis that controls for this overlap, would strengthen the claim that the method yields general lexical-semantic understanding rather than task-specific alignment.
minor comments (4)
  1. [Abstract and Table 3] The abstract reports a WiC score of 72.14 while Table 3 reports 72.1; the decimal places should be aligned.
  2. [Throughout] The paper contains several typos and spelling errors, including 'explicitely', 'tokanization', 'rarley', 'pracrical', 'Tranformer', and 'bi-product'; a careful proofreading pass is needed.
  3. [Figure 3] The qualitative examples in Figure 3 are anecdotal; adding a short quantitative analysis (e.g., agreement with SemCor labels on a sample) would make the demonstration more informative.
  4. [Section 3.4, Table 1] The 'average embedding' variant is evaluated only on SemEval-SS; it would be useful to know whether that variant also matches the 60K vocabulary model on WiC and GLUE before concluding that the two rare-word handling methods are comparable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the pretraining objective and the external evaluation benchmarks are distinct, and the overlapping WordNet supersense taxonomy is task alignment, not a derivation from the paper's own inputs.

full rationale

SenseBERT's central claims are tested on external benchmarks: SemEval-SS is a supersense-converted version of the SemCor/SenseEval/SemEval WSD suites standardized by Raganato et al., and WiC is from the external SuperGLUE benchmark. Neither benchmark's labels are used to set pretraining constants or to train the auxiliary supersense head; the human-annotated gold labels are independent of the pretraining signal. The paper's use of WordNet to define allowed supersenses A(w) for the masked-word objective does mean that the pretraining label space matches the coarse-grained evaluation taxonomy, but that is an intentional transfer setup rather than a circular reduction: the model never sees SemEval-SS or WiC labels during pretraining, and the evaluation measures whether the learned representations transfer to annotated examples. No parameter is fitted to the evaluation subsets and then reported as a prediction. There are no load-bearing self-citations; the cited models and baselines (BERT, RoBERTa, KnowBERT, ELMo, and the sense-embedding work) are external, and the paper invokes no uniqueness theorem from the authors' own prior work. The absence of an ablation that isolates the LSLM loss from the 60K vocabulary, the single-supersense masking curriculum, and the S·M input term is a real attribution or confound concern, but it is not circular because the reported improvements do not reduce by construction to the pretraining objective's own inputs. Similarly, the observation that Eq. 7 is minimized per instance by a uniform distribution over A(w) independently of context is a serious effectiveness concern about whether the loss actually teaches context-dependent supersense selection, but it is a correctness issue, not a circularity. Thus the derivation chain remains self-contained with respect to external evidence, and the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim does not introduce new theoretical entities or fitted parameters beyond standard hyperparameters. The load is carried by the domain assumption that WordNet supersenses are a useful weak-supervision signal, and by the experimental evidence on external benchmarks.

free parameters (4)
  • single_supersense_mask_fraction = 0.5
    Hand-chosen fraction of masked words that are single-supersensed, intended to give unambiguous semantic signal; not fitted to downstream task, but affects the training distribution.
  • single_supersense_budget_cap = 0.4
    Hand-chosen cap on the fraction of the 15% masking budget filled by single-supersensed words; a design choice that could interact with learning.
  • supersense_loss_weights = 1.0 and 1.0
    The allowed-senses and regularization losses are combined with equal weight in equation (7), with no reported tuning; the weighting affects the balance between fitting allowed senses and preventing collapse.
  • vocabulary_size = 60K
    The 60K-token vocabulary is a hand-chosen augmentation of BERT's 30K vocabulary to increase coverage of whole words; it adds parameters and could affect behavior.
assumptions (4)
  • domain assumption WordNet supersense categories are a meaningful and transferable coarse-grained semantic classification.
    The whole method and the SemEval-SS evaluation are built on this taxonomy; if the 45 supersenses are not well-aligned with how meaning is used in text, the training signal is weak.
  • domain assumption The soft-labeling scheme over all allowed supersenses yields approximately correct sense probabilities given sufficient corpus diversity.
    Section 3.2 argues that repeated contexts will reinforce the true sense and cancel out noise; this assumes the corpus is balanced enough.
  • domain assumption The WiC dataset labels are consistent with WordNet supersense distinctions.
    The paper states WiC is defined over supersenses, so the benchmark measures the same granularity the model is pretrained on; if the mapping is imperfect, WiC numbers are harder to interpret.
  • standard math The BERT architecture and its pre-training recipe are taken as background and transfer to the new objective without structural changes.
    Sections 3.1 and 3.5 inherit BERT's architecture, masking ratio, and optimizer details from Devlin et al.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SenseBERT: Driving Some Sense into BERT." pith.science (2026). https://pith.science/paper/A3ELOQPD

@misc{pith2026190805646,
  author       = {Pith},
  title        = {Pith review of: SenseBERT: Driving Some Sense into BERT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A3ELOQPD}},
  note         = {Machine review of arXiv:1908.05646}
}
read the original abstract

The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet supersenses. Accordingly, we attain a lexical-semantic level language model, without the use of human annotation. SenseBERT achieves significantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the Word in Context task.

Figures

Figures reproduced from arXiv: 1908.05646 by the authors.

Figure 1
Figure 1. SenseBERT includes a masked-word supersense prediction task, pre-trained jointly with BERT’s original [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. UMAP visualization of supersense vectors (rows of the classifier [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. (a) A demonstration of supersense probabilities assigned to a masked position within context, as given by SenseBERT’s word-supersense level semantic language model (capped at 5%). Example words corresponding to each supersense are presented in parentheses. (b) Examples of SenseBERT’s prediction on raw text, when the unmasked input sentence is given to the model. This beyond word-form abstraction ability facilitates … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Example entries of (a) the SemEval-SS task, where a model is to predict the supersense of the marked word, and (b) the Word in Context (WiC) task where a model must determine whether the underlined word is used in the same/different supersense within sentences A and B.…
Figure 5
Figure 5. Figure 5: Dendrogram visualization of an Agglomerative hierarchical clustering over the supersense vectors (rows [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 18 canonical work pages

  1. [1]

    Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. https://doi.org/10.1162/tacl_a_00034 Linear algebraic structure of word senses, with applications to polysemy . Transactions of the Association for Computational Linguistics, 6:483--495

    doi:10.1162/tacl_a_00034 2018

  2. [2]

    Pierpaolo Basile. 2012. Super-sense tagging using support vector machines and distributional features. In International Workshop on Evaluation of Natural Language and Speech Tool for Italian, pages 176--185. Springer

    work page 2012

  3. [3]

    William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. 2019. https://arxiv.org/abs/1906.01604 KERMIT : Generative insertion-based modeling for sequences . arXiv preprint arXiv:1906.01604

    arXiv 2019

  4. [4]

    Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. https://doi.org/10.3115/v1/D14-1110 A unified model for word sense representation and disambiguation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 1025--1035, Doha, Qatar. Association for Computational Linguistics

    doi:10.3115/v1/d14-1110 2014

  5. [5]

    Massimiliano Ciaramita and Mark Johnson. 2003. https://www.aclweb.org/anthology/W03-1022 Supersense tagging of unknown nouns in W ord N et . In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 168--175

    work page 2003

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  7. [7]

    Philip Edmonds and Scott Cotton. 2001. https://www.aclweb.org/anthology/S01-1001 SENSEVAL -2: Overview . In Proceedings of SENSEVAL -2 Second International Workshop on Evaluating Word Sense Disambiguation Systems , pages 1--5, Toulouse, France. Association for Computational Linguistics

    work page 2001

  8. [8]

    John Hewitt and Christopher D. Manning. 2019. https://doi.org/10.18653/v1/N19-1419 A structural probe for finding syntax in word representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4129--4138, Minneapol...

    doi:10.18653/v1/n19-1419 2019

Show all 41 references
  1. [9]

    Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. https://doi.org/10.18653/v1/P16-1085 Embeddings for word sense disambiguation: An evaluation study . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2016 doi

  2. [10]

    Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. https://openreview.net/pdf?id=r1aPbsFle Tying word vectors and word classifiers: A loss framework for language modeling . In ICLR

    2017

  3. [11]

    Adam Kilgarriff. 1997. I don’t believe in word senses. Computers and the Humanities, 31(2):91--113

    1997

  4. [12]

    Minh Le, Marten Postma, Jacopo Urbani, and Piek Vossen. 2018. https://www.aclweb.org/anthology/C18-1030 A deep dive into word sense disambiguation with LSTM . In Proceedings of the 27th International Conference on Computational Linguistics, pages 354--365, Santa Fe, New Mexico...

    2018

  5. [13]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 RoBERTa : A robustly optimized bert pretraining approach . arXiv preprint arXiv:1907.11692

    2019 arXiv

  6. [14]

    Daniel Loureiro and Al \' pio Jorge. 2019. https://doi.org/10.18653/v1/P19-1569 Language modelling makes sense: Propagating representations through W ord N et for full-coverage word sense disambiguation . In Proceedings of the 57th Annual Meeting of the Association for Computa...

    2019 doi

  7. [15]

    Leland McInnes, John Healy, and James Melville. 2018. https://arxiv.org/abs/1802.03426 UMAP : Uniform manifold approximation and projection for dimension reduction . arXiv preprint arXiv:1802.03426

    2018 arXiv

  8. [16]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf Distributed representations of words and phrases and their compositionality . In Advan...

    2013

  9. [17]

    George A Miller. 1998. WordNet: An electronic lexical database. MIT press

    1998

  10. [18]

    Miller, Claudia Leacock, Randee Tengi, and Ross T

    George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. https://www.aclweb.org/anthology/H93-1061 A semantic concordance . In H uman L anguage T echnology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993

    1993

  11. [19]

    Andrea Moro and Roberto Navigli. 2015. https://doi.org/10.18653/v1/S15-2049 S em E val-2015 task 13: Multilingual all-words sense disambiguation and entity linking . In Proceedings of the 9th International Workshop on Semantic Evaluation ( S em E val 2015) , pages 288--297, De...

    2015 doi

  12. [20]

    Roberto Navigli. 2009. https://doi.org/10.1145/1459352.1459355 Word sense disambiguation: A survey . ACM Comput. Surv., 41(2)

    2009

  13. [21]

    Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. https://www.aclweb.org/anthology/S13-2040 S em E val-2013 task 12: Multilingual word sense disambiguation . In Second Joint Conference on Lexical and Computational Semantics (* SEM ), Volume 2: Proceedings of the Seve...

    2013

  14. [22]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. https://doi.org/10.3115/v1/D14-1162 G love: Global vectors for word representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 1532--1543, Doha,...

    2014 doi

  15. [23]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North A merican Chapter of the Assoc...

    2018 doi

  16. [24]

    Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A

    Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. https://doi.org/10.18653/v1/D19-1005 Knowledge enhanced contextual word representations . In Proceedings of the 2019 Conference on Empirical Methods in Natural Lang...

    2019 doi

  17. [25]

    Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. https://doi.org/10.18653/v1/N19-1128 W i C : the word-in-context dataset for evaluating context-sensitive meaning representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association ...

    2019 doi

  18. [26]

    Sameer Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007. https://www.aclweb.org/anthology/S07-1016 S em E val-2007 task-17: E nglish lexical sample, SRL and all words . In Proceedings of the Fourth International Workshop on Semantic Evaluations ( S em E val-2007...

    2007

  19. [27]

    Ofir Press and Lior Wolf. 2017. https://www.aclweb.org/anthology/E17-2025 Using the output embedding to improve language models . In Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Linguistics: Volume 2, Short Papers , pages 157...

    2017

  20. [28]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  21. [29]

    Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017. https://www.aclweb.org/anthology/E17-1010 Word sense disambiguation: A unified evaluation framework and empirical comparison . In Proceedings of the 15th Conference of the E uropean Chapter of the Associati...

    2017

  22. [30]

    Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. http://papers.nips.cc/paper/9065-visualizing-and-measuring-the-geometry-of-bert.pdf Visualizing and measuring the geometry of BERT . In Advances in Neural Information Proc...

    2019

  23. [31]

    Sascha Rothe and Hinrich Sch \"u tze. 2015. https://doi.org/10.3115/v1/P15-1173 A uto E xtend: Extending word embeddings to embeddings for synsets and lexemes . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International...

    2015 doi

  24. [32]

    Nathan Schneider. 2014. Lexical semantic analysis in natural language text. Unpublished Doctoral Dissertation, Carnegie Mellon University

    2014

  25. [33]

    Nathan Schneider and Noah A. Smith. 2015. https://doi.org/10.3115/v1/N15-1177 A corpus and model integrating multiword expressions and supersenses . In Proceedings of the 2015 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Lan...

    2015 doi

  26. [34]

    Benjamin Snyder and Martha Palmer. 2004. https://www.aclweb.org/anthology/W04-0811 The E nglish all-words task . In Proceedings of SENSEVAL -3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text , pages 41--43, Barcelona, Spain. Ass...

    2004

  27. [35]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention is all you need . In Advances in Neural Information Processing Systems 30,...

    2017

  28. [36]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. http://papers.nips.cc/paper/8589-superglue-a-stickier-benchmark-for-general-purpose-language-understanding-systems.pdf SuperGLUE : A stickier benchmar...

    2019

  29. [37]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  30. [38]

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. http://papers.nips.cc/paper/8812-xlnet-generalized-autoregressive-pretraining-for-language-understanding.pdf XLNet : Generalized autoregressive pretraining for language understand...

    2019

  31. [39]

    Dayu Yuan, Julian Richardson, Ryan Doherty, Colin Evans, and Eric Altendorf. 2016. https://www.aclweb.org/anthology/C16-1130 Semi-supervised word sense disambiguation with neural models . In Proceedings of COLING 2016, the 26th International Conference on Computational Linguis...

    2016

  32. [40]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.