{"id":"eae9adae-4d77-4d2f-9e76-f637b099a26f","arxiv_id":"1908.05646","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding a masked-word supersense prediction task to BERT's pre-training, with WordNet supersenses as weak labels, improves performance on supersense disambiguation and Word in Context without human sense-annotated data.","lead":"SenseBERT trains BERT to also predict the WordNet supersense of masked words during pre-training, using unlabeled text and WordNet as weak supervision. The resulting model beats BERT on supersense disambiguation and set a state-of-the-art score on the Word in Context benchmark at publication time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation isolates the auxiliary supersense loss from the 60K vocabulary, single-supersense masking, and sense-augmented input embeddings; reported gains may stem from confounds rather than the paper's core novelty.","rationale":"The most load-bearing uncertainty is causal attribution. The reader's soft-label-noise concern is real but secondary: even if the noise-cancellation argument is imperfect, the method could still improve lexical understanding through other channels. Before worrying about rare-sense bias, one must establish that the supersense loss contributes at all. The paper's comparisons hold no component fixed: vocabulary size, masking distribution, and input construction all change together. The LSLM objective's tendency to favor uniform distributions over A(w) for a known word (Eqs. 4-7) makes it plausible that the sense head itself contributes little to context-dependent disambiguation; much of the benefit could come from the single-supersense masking curriculum or the static sense prior in Eq. 8. A single ablation (zeroing LSLM) would settle this. If the no-LSLM baseline reproduces the gains, the central claim is not supported; if it does not, the claim survives. The reader's conditional verdict is appropriate; I would make the ablation an explicit acceptance condition.","tokens_in":12992,"tokens_out":19990,"duration_ms":214499,"concrete_test":"Pretrain a BERTBASE control with the exact SenseBERT setup—same pretraining data, 60K vocabulary, same single-supersense-prioritized masking, and same S·M input injection—but with the supersense loss LSLM set to zero (i.e., only MLM+NSP). Run the same SemEval-SS Frozen and WiC evaluations. As a second arm, keep LSLM but remove the S·M input injection to separate the loss from the embedding prior. If the no-LSLM arm matches SenseBERT within about one point, the supersense objective is not the driver of the claimed gains; if it drops substantially, the objective is load-bearing.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that the auxiliary masked-word supersense objective with WordNet allowed senses as soft weak labels is what improves lexical understanding. But the published comparisons do not isolate that objective. SenseBERT BASE differs from the BERTBASE baseline in at least three confounded ways: (i) a 60K-token vocabulary vs BERT's 30K (Section 3.4); (ii) a masking curriculum that deliberately over-samples single-supersense content words (Section 3.5); and (iii) static supersense vectors injected into the input through S·M in Eq. 8. Tables 2 and 3 compare only the full combination against vanilla BERT; no arm removes LSLM (Eq. 7) while keeping the other components. The LSLM loss itself has a property that makes this confound especially important: for a fixed masked word w, L_allowed + L_reg is minimized by a uniform distribution over A(w) independent of context, because L_reg penalizes concentration on any subset of allowed supersenses. Context-sensitivity can only enter indirectly, through averaging over different masked words that share a surface context. It is therefore not safe to attribute the reported SemEval-SS and WiC gains to the sense-prediction objective; they could come from better tokenization, the masking curriculum, or the static sense prior in the input.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SenseBERT, a BERT-style model pretrained with an auxiliary masked-word WordNet supersense prediction objective. The model is trained to predict both the masked word and its supersenses, using a soft-labeling scheme over WordNet's allowed supersenses, and it augments input embeddings with static supersense vectors. The authors also enlarge the vocabulary to 60K tokens and bias the masking strategy toward single-supersense words. They evaluate on SemEval-SS, a supersense version of the SemEval WSD benchmark, and on WiC from SuperGLUE, reporting that SenseBERT BASE outperforms BERTBASE and BERTLARGE on SemEval-SS and that SenseBERT LARGE achieves 72.14 on WiC, a state-of-the-art score at the time. They also report GLUE results showing no overall degradation relative to a BERTBASE baseline trained on the same pretraining data.","tokens_in":13264,"tokens_out":3885,"duration_ms":39879,"significance":"If the reported gains are attributable to the sense-level pretraining objective, this is a meaningful contribution: it demonstrates that weak lexical supervision from WordNet can be injected at scale without human annotation, and the frozen-embedding probe improvements in Table 2 are particularly striking. The authors also deserve credit for training a BERTBASE baseline on the same pretraining data for GLUE, which controls for corpus differences. However, the paper does not isolate the proposed auxiliary loss from other architectural and training changes, and both main evaluation tasks share the same supersense label space used in pretraining. The significance of the central claim is therefore conditional on additional evidence that the gains come from the sense-prediction objective itself.","major_comments":[{"comment":"No ablation isolates the auxiliary supersense loss. SenseBERT BASE differs from BERTBASE in at least three ways: the 60K-token vocabulary versus BERT's 30K (Section 3.4), the single-supersense-biased masking strategy (Section 3.5), and the sense-augmented input embeddings of Eq. 8 (Section 3.3). The reported comparisons compare only the full combination against vanilla BERT; an arm that keeps the same vocabulary, masking, and input embeddings but removes the LSLM loss (Eq. 7) is necessary to attribute the SemEval-SS and WiC gains to the sense-prediction objective. Without such an ablation, the central claim is underdetermined.","section":"Section 3.2-3.5; Tables 2-3"},{"comment":"The combined loss has a context-independent minimizer: for a fixed masked word w, LSLM is minimized by the uniform distribution over the allowed supersenses A(w), because L_allowed is maximized when the total probability on A(w) is 1, while L_reg penalizes any non-uniformity within A(w). Context sensitivity can only enter indirectly, through averaging over different masked words that share a context. The paper should address this property explicitly or provide evidence that the model's supersense predictions are context-sensitive as a direct result of LSLM rather than of the word-form LM or the input embeddings.","section":"Section 3.2, Eqs. (4)-(7)"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported, so it is unclear whether differences of 1-3 points (e.g., 70.3 versus 69.6 on WiC, 77.9 versus 77.5 on GLUE) are significant. The claim of 'significantly improved' lexical understanding needs variance estimates or at least repeated fine-tuning runs. In addition, the BERTLARGE WiC baseline is taken from the published SuperGLUE score rather than re-evaluated under the same fine-tuning protocol as SenseBERT, which weakens the comparability of the WiC comparison.","section":"Sections 5.2-5.3; Tables 2-4"},{"comment":"WiC's label scheme is deliberately defined over WordNet supersenses, the same 45-category inventory that SenseBERT is pretrained to predict. This alignment can inflate the apparent improvement: the model is trained to output exactly the label space used to define positive and negative WiC pairs. Evaluating on a task whose labels do not coincide with the pretraining taxonomy, or performing an analysis that controls for this overlap, would strengthen the claim that the method yields general lexical-semantic understanding rather than task-specific alignment.","section":"Section 5.3; Table 3"}],"minor_comments":[{"comment":"The abstract reports a WiC score of 72.14 while Table 3 reports 72.1; the decimal places should be aligned.","section":"Abstract and Table 3"},{"comment":"The paper contains several typos and spelling errors, including 'explicitely', 'tokanization', 'rarley', 'pracrical', 'Tranformer', and 'bi-product'; a careful proofreading pass is needed.","section":"Throughout"},{"comment":"The qualitative examples in Figure 3 are anecdotal; adding a short quantitative analysis (e.g., agreement with SemCor labels on a sample) would make the demonstration more informative.","section":"Figure 3"},{"comment":"The 'average embedding' variant is evaluated only on SemEval-SS; it would be useful to know whether that variant also matches the 60K vocabulary model on WiC and GLUE before concluding that the two rare-word handling methods are comparable.","section":"Section 3.4, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the same-data BERT baseline for GLUE is a good control. The main barrier is the missing ablation: the paper's headline claim requires attributing the gains to the LSLM objective, but the current design changes several factors at once. The formal property of Eq. 7 also deserves a direct response. I would be willing to reconsider after the ablation and a discussion of the WiC label-space alignment are added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper adds a supersense prediction head to BERT's masked-word objective, using WordNet's allowed-sense lists as soft weak labels. That is new relative to KnowBERT, Loureiro and Jorge, and vanilla BERT, and the writing is honest about which parts are engineering choices (60K vocab, single-supersense masking). The frozen-probe result on SemEval-SS is a genuinely nice way to show the pretrained representation carries sense information.\n\nThe problem is that the headline claim—that the auxiliary sense loss is what drives the gains—is not actually tested. SenseBERT differs from the published BERT baselines in at least three ways: vocabulary size (60K vs 30K), the masking curriculum that oversamples single-supersense words, and the static supersense embeddings added to the input (Eq. 8). Table 1 compares 30K vs 60K and OOV handling, but no arm removes LSLM itself. Worse, the loss in Eq. 7 has a mathematical property that should worry any referee: for a fixed masked word w, L_allowed + L_reg is minimized by a uniform distribution over A(w), independent of context. Context can only enter through averaging over words that share the context. That is a real flaw in the design, and it means the improvements could plausibly come from the other components.\n\nThe WiC result looks less surprising once you recall that WiC's labels are defined over supersenses—the same granularity the model is pretrained to predict. That's not circular, but it is a close alignment. There's also no ablation of the pretraining corpus for the lexical tasks: the BERT baselines on SemEval-SS and WiC are the published models, whereas the GLUE baseline is retrained on the same data. So the 10-point frozen gap on SemEval-SS could partly be a data effect, not just the sense signal.\n\nThe paper is worth taking seriously. The idea is simple, the write-up is clear, and they give a GLUE comparison that shows no regression. But the central claim is underdetermined by the experiments. A referee should ask for an ablation that removes LSLM while keeping the other modifications, or at least a discussion of why that's not feasible. The loss issue deserves a theoretical note as well. Give it a careful referee slot, but expect heavy revision.","headline":"Good idea, weakly supported: the sense-prediction loss is novel but never isolated from vocabulary, masking, and input-embedding changes, and the loss itself is mathematically biased toward uniform predictions over allowed senses.","tokens_in":13833,"tokens_out":4540,"would_cite":false,"duration_ms":45409,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training a language model to predict word supersenses improves lexical understanding.","keywords":["word sense disambiguation","supersense tagging","BERT","weak supervision","WordNet","lexical semantics","Word in Context","language model pretraining"],"falsifier":"Train SenseBERT on a corpus from which one common sense of a polysemous word such as 'bass' as a fish has been removed, then evaluate on WiC and SemEval-SS items that require that sense; if accuracy on those items collapses while other-sense items stay high, the soft-label cancellation mechanism is not doing what the paper claims.","tokens_in":12785,"feed_emoji":"🧠","tokens_out":5498,"duration_ms":48806,"temperature":0.7,"pith_summary":"SenseBERT claims that a language model can be taught word meanings during pre-training without any human sense annotations by adding a second prediction task: alongside guessing a masked word's identity, the model must guess its WordNet supersense, meaning its coarse semantic class such as noun.food or verb.creation. The training signal is weak, because every supersense WordNet allows for the masked word is treated as a possible label, but the paper argues that across a large corpus the correct sense is consistently reinforced while the competing labels cancel out. The result is that the model's representations carry more extractable lexical semantics: a frozen SenseBERT outperforms much larger plain BERT models on supersense disambiguation, and the largest version sets a then-state-of-the-art score on the Word in Context task, which asks whether a word is used in the same sense in two different contexts. This matters because it shows a path to injecting semantic knowledge into self-supervised language models using only an existing lexical database.","feed_headline":"Giving BERT word-sense targets beats a bigger BERT","feed_subtitle":"An auxiliary WordNet supersense objective lifts lexical understanding without human labels and sets a WiC record.","key_machinery":"The load-bearing mechanism is a parallel external mapping $S \\in \\mathbb{R}^{d \\times 45}$ from the transformer's hidden states to the 45 WordNet supersense categories, trained jointly with BERT's word mapping $W$. For each masked word, the loss combines an allowed-senses term $-\\log \\sum_{s \\in \\mathcal{A}(w)} p(s \\mid \\text{context})$ with a regularization term that keeps the predicted distribution close to uniform over the allowed senses, and the same matrix $S$ is also added to the input embeddings through a fixed word-to-supersenses 0/1 matrix $M$, so the input vector becomes $v_{\\text{input}} = (W + SM)x + p$. The supersense signal thus shapes the representations the model sees as well as the predictions it is scored on.","core_discovery":"On the paper's own terms, the central discovery is that adding a weakly supervised word-supersense prediction objective to BERT's masked-language-model pre-training produces a lexical-semantic level language model: the same transformer that predicts word forms also learns to predict meaning categories. Concretely, SenseBERT BASE scores 75.6 frozen and 83.0 fine-tuned on the SemEval-SS supersense disambiguation benchmark, versus 65.1 and 79.2 for BERT BASE, while SenseBERT LARGE reaches 72.14 on the Word in Context task, surpassing BERT LARGE at 69.6 and also prior WordNet-infused models. The paper interprets these gains as evidence that word-form-level self-supervision alone leaves a measurable gap in lexical semantics, and that a sense-level auxiliary task can largely close that gap without human annotation.","pith_inferences":["Beyond the paper: the soft-label cancellation assumption could be stress-tested by constructing corpora with artificially skewed sense frequencies and measuring whether rare senses stop being learned.","Beyond the paper: because the supersense signal is word-level and does not depend on the surface form, the same recipe could be applied to other lexical inventories or to languages that lack sense-annotated corpora.","Beyond the paper: the enlarged 60K vocabulary adds 23 to 30 million parameters over vanilla BERT, so a controlled comparison using a same-size vocabulary without the sense head would isolate how much of the WiC gain comes from sense supervision rather than added capacity."],"forward_implications":["Supersense disambiguation on SemEval-SS improves by more than 10 points in the frozen-embedding setting and by about 4 points after fine-tuning for both BASE and LARGE model sizes.","A single SenseBERT LARGE model achieves 72.14 on the Word in Context task, which at publication time was the state of the art and 2.5 points above BERT LARGE.","Lexical-semantic gains do not come at the cost of general understanding: SenseBERT BASE scores 77.9 on GLUE, on par with a same-data BERT BASE at 77.5.","The learned supersense matrix organizes the 45 supersenses into meaningful clusters, giving the model an explicit sense-level view of raw text during pre-training."],"supporting_citations":[{"why":"Provides the BERT architecture and masked-language-model pre-training objective that SenseBERT extends with a supersense prediction head.","marker":"Devlin et al., 2019"},{"why":"Supplies the WordNet inventory from which each word's allowed supersenses are extracted.","marker":"Miller, 1998"},{"why":"Defines the standardized WSD evaluation framework and data that the paper adapts into the SemEval-SS benchmark.","marker":"Raganato et al., 2017"},{"why":"Introduces the Word in Context dataset used to measure supersense awareness.","marker":"Pilehvar and Camacho-Collados, 2019"},{"why":"The SuperGLUE benchmark that hosts WiC and provides the baseline BERT LARGE score SenseBERT is compared against.","marker":"Wang et al., 2019"},{"why":"A prior WordNet-infused contextual representation model that serves as a WiC baseline SenseBERT surpasses.","marker":"Peters et al., 2019"},{"why":"A sense-embedding baseline built from BERT and WordNet that SenseBERT compares against on WiC.","marker":"Loureiro and Jorge, 2019"}],"fun_headline_variants":["Teaching BERT word senses beats a bigger BERT","SenseBERT: Word-sense pre-training beats scaling BERT","No labels needed: BERT learns word senses, sets WiC record","Weakly supervised word senses give BERT a semantic boost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that, over a large unlabeled corpus, the correct supersense of a masked word is reinforced by its context more often than the other supersenses WordNet allows, so the noise from wrong soft labels averages out; if corpus sense frequencies are skewed, the model could systematically strengthen incorrect supersenses.","fun_headline_variants_meta":{"raw":{"variants":["Teaching BERT word senses beats a bigger BERT","SenseBERT: Word-sense pre-training beats scaling BERT","No labels needed: BERT learns word senses, sets WiC record","Weakly supervised word senses give BERT a semantic boost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1296,"prompt_tokens":838,"completion_tokens":458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":388}},"tokens_in":454,"tokens_out":458,"duration_ms":5048,"temperature":1.0,"reasoning_tokens":388,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:06:42.804419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SenseBERT on a corpus from which one common sense of a polysemous word such as 'bass' as a fish has been removed, then evaluate on WiC and SemEval-SS items that require that sense; if accuracy on those items collapses while other-sense items stay high, the soft-label cancellation mechanism is not doing what the paper claims.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the WordNet inventory from which each word's allowed supersenses are extracted."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the standardized WSD evaluation framework and data that the paper adapts into the SemEval-SS benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The SuperGLUE benchmark that hosts WiC and provides the baseline BERT LARGE score SenseBERT is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A sense-embedding baseline built from BERT and WordNet that SenseBERT compares against on WiC."}],"review_version":1}