{"id":"4127c1b2-923b-4c24-8b35-e3b810cf6aa5","arxiv_id":"1909.02339","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LIBERT, a BERT variant pretrained with an auxiliary word-pair similarity task, outperforms BERT on 9/10 GLUE tasks and on three lexical simplification datasets.","lead":"The authors augment BERT's pretraining with a third task that classifies word pairs from lexical resources like WordNet, creating a model called LIBERT. On the GLUE benchmark, LIBERT beats the original BERT on 9 of 10 tasks and improves lexical simplification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No matched-control comparison: LIBERT receives 2M extra gradient updates and large extra constraint data, so the reported gains cannot yet be attributed to lexico-semantic knowledge.","rationale":"The paper provides internally consistent, directionally positive results across GLUE and lexical simplification, and it is careful to train both models from scratch with the same base architecture. The central weakness is causal attribution: the alternating training regime means LIBERT is also receiving substantially more gradient updates and more training data than BERT. The authors say the 'only difference' is the extra LRC loss, but that is true only at the level of objectives, not at the level of optimization budget. Because no compute-matched BERT baseline and no non-lexical auxiliary-task control are reported, the observed gains may reflect extra training signal rather than the semantic content of the WordNet/Roget constraints. This concern is load-bearing because the paper's title, abstract, and conclusion claim specifically that lexico-semantic knowledge is beneficial, not merely that additional pretraining-style updates help. The reader's stated weakest assumption about generalization versus memorization is plausible but not the most direct threat; even a fully generalizing LRC classifier would not establish the causal role of lexical semantics without a matched control. The absence of released code and models, noted by the reader, further prevents checking whether the comparison was fair. Given these unresolved confounds, the reader's CONDITIONAL verdict remains appropriate; the condition should explicitly require a matched-control experiment before the central attribution is accepted.","tokens_in":13827,"tokens_out":4811,"duration_ms":56090,"concrete_test":"Train a BERT control model with the identical alternating schedule and constraint-format inputs, but replace the WordNet/Roget relation labels with randomly shuffled binary labels (or with arbitrary word-pair classification labels); match LIBERT's total number of updates. Evaluate on the GLUE dev sets at 1M and 2M reported MLM steps. If the control reproduces LIBERT's gains within about one point, the reported advantage is not attributable to semantic similarity. A cheaper secondary check is to train vanilla BERT for twice as many MLM+NSP steps and see whether the gap closes at equal total update count.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 4.1, LIBERT is trained with an alternating regime: one MLM+NSP batch update followed by one LRC batch update. Therefore each reported 'MLM+NSP step' for LIBERT is accompanied by an additional LRC update, while BERT receives only the MLM+NSP update. At 2M reported steps, LIBERT has consumed 2M LRC batches (each with k positive and 2k negative word pairs) on top of 2M sentence batches; BERT has consumed only the 2M sentence batches. The two models thus differ in total optimizer steps, total data volume, and total gradient signal, not solely in the presence of lexical knowledge. The paper reports no BERT baseline trained for the same total number of updates and no control auxiliary task with the same training schedule but non-lexical or label-shuffled constraints. Consequently, the central causal claim that semantic-similarity constraints are responsible for the GLUE and lexical-simplification gains is not isolated from a generic 'more pretraining data and compute helps' effect. The reader's concern about memorization versus generalization is related, but even a perfectly generalizing LRC objective could fail to be the cause if the extra optimization budget alone reproduces the improvements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LIBERT, a variant of BERT that adds a third pretraining objective, Lexical Relation Classification (LRC), to the standard masked language modeling and next sentence prediction tasks. The LRC objective is trained on roughly 1.35M synonym and direct hyponym-hypernym pairs from WordNet and Roget's Thesaurus, with negatives constructed from in-batch nearest neighbors in a fastText space. The authors train LIBERT and a vanilla BERT baseline from scratch under the same configuration, evaluate on the GLUE benchmark and on three lexical simplification datasets, and report that LIBERT outperforms BERT on 9/10 GLUE test tasks (and ties on QNLI) after 2M reported MLM+NSP steps, with the largest margins on CoLA, AX, and SST-2, and that it improves lexical simplification on all three benchmarks with gains up to 8.2 points. They also present a fine-grained analysis on the GLUE Diagnostic dataset showing larger gains in categories such as Lexical Semantics, Logic, and Knowledge & Common Sense.","tokens_in":1936,"tokens_out":1657,"duration_ms":37271,"significance":"If the reported gains are causally attributable to the injected lexical knowledge, the paper would make a useful contribution: it demonstrates a simple, general way to complement distributional pretraining with structured lexical resources and shows downstream benefits beyond tasks where word-level similarity is an obvious ingredient. Strengths of the work include the design decision to train both models from scratch with the same hyperparameters, the use of external benchmarks that are not used to fit the model, the diagnostic breakdown on GLUE's Diagnostic set, and the fact that gains persist after 2M steps. The lexical simplification experiments are also a sensible similarity-focused testbed. However, the central causal claim is not yet established because the comparison is not matched in total optimization budget, and the reported differences on several GLUE tasks are within typical run-to-run variance for these benchmarks. The core idea is promising and the evidence is suggestive, but the current manuscript needs a stronger experimental design or a more modest interpretation before the attribution claim can be accepted.","major_comments":[{"comment":"The experimental comparison is confounded by an unmatched training budget. The paper states that LIBERT updates parameters in an alternating regime: one batch of MLM+NSP followed by one batch of LRC. Therefore, after the reported '2M MLM+NSP steps,' LIBERT has additionally consumed 2M LRC batches, each containing k positive and 2k negative word pairs, while vanilla BERT has received only the 2M sentence batches. The two models thus differ not only in the presence of lexical constraints but also in total optimizer steps, total data volume, and total gradient signal. The central claim that lexico-semantic knowledge causes the observed gains therefore requires additional controls, such as (a) a BERT baseline trained for the same total number of updates with an auxiliary task on shuffled or non-lexical constraint pairs, or (b) a BERT baseline trained for the same total number of updates with an auxiliary task of comparable difficulty that contains no semantic-similarity information. Without such a control, the reported gains could plausibly reflect additional compute and data rather than semantic specialization.","section":"Section 4.1, Training and Evaluation"},{"comment":"No variance or significance information is reported, and several differences are small enough to be attributable to random seed variation. For example, at 2M steps the test-set differences are +0.2 on RTE, +0.3 on QQP F1, +0.7 on MNLI-mm, and +0.0 on QNLI; single-run differences of this magnitude are typically within the run-to-run noise for GLUE tasks. The paper should report means and standard deviations over multiple random seeds (at least for fine-tuning, and preferably also for pretraining or with several pretraining seeds), and indicate whether the differences are statistically significant. This issue is load-bearing because the headline claim is an improvement on '9 out of 10' tasks, many of which rest on small single-run margins.","section":"Section 4.2, Table 2"},{"comment":"The fine-grained Diagnostic analysis reports very large relative gains on small subsets (e.g., Factivity +281.7% at 1M steps), but no information is given about the number of examples in each fine-grained category or the variance of the R3 coefficient. Large percentage changes on small subsets should be interpreted cautiously. This is not the central claim of the paper, but the analysis as presented may overstate the reliability of the diagnostic conclusions.","section":"Section 4.2, Table 3"},{"comment":"The lexical simplification gains are also reported as single-run numbers without variance or significance testing. In Table 4, the differences in accuracy are sometimes substantial (e.g., +8.2 on LexMTurk at 1M steps), but the precision/recall/F1 differences in candidate generation are often small (e.g., +0.0098 recall on LexMTurk at 1M steps). The claim that LIBERT 'consistently outperforms BERT' would be strengthened by multiple seeds and a statement of significance, especially because the pipeline includes additional features such as fastText similarity and word frequency that are held fixed across models.","section":"Section 5, Results and Discussion"}],"minor_comments":[{"comment":"There is a typo 'the the original work' in the footnote about batch size. The footnote itself is useful context, but the sentence should be corrected.","section":"Section 4.1, Training and Evaluation"},{"comment":"The column header 'accurracy' is misspelled; it should be 'accuracy'.","section":"Table 4"},{"comment":"The notation WLRC∈RH×2 and bLRC∈R2 is fine, but it would be clearer to state explicitly that the LRC classifier is used only during pretraining and is discarded or re-initialized during downstream fine-tuning. The text implies this, but an explicit statement would avoid ambiguity.","section":"Section 3.2, Eq. (1)"},{"comment":"The description of the BERT-LS approach is clear, but the paper should specify whether the LRC classifier is used at all during lexical simplification inference, since only the MLM predictor appears to be used. Clarifying this would help readers understand exactly which components differ between BERT and LIBERT at evaluation time.","section":"Section 5, Experimental Setup"},{"comment":"The two 'ERNIE' references (Zhang et al., 2019a and 2019b) appear to have very similar titles and one appears twice with different arXiv numbers; the authors should check whether these are meant to be distinct papers and cite them appropriately.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a 2019/2020 submission and the experimental confound in Section 4.1 is the main obstacle. The reported gains may still be real, but the manuscript cannot support the causal attribution as written. I would ask for a matched-control experiment or a substantially softened interpretation of the results. The paper's framing in the broader literature is reasonable, and I do not see a novelty or scope problem; the issue is experimental rigor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nYou should know two things about arXiv:1909.02339. The idea is decent: it is the first attempt in this line of work to inject word-level lexical constraints into BERT via an auxiliary relation classification objective, and the evaluation on lexical simplification is well chosen. But the headline claim—that lexical knowledge causes the gains—is not actually tested. The training regime alternates one MLM+NSP update with one LRC update, so after 2M reported steps LIBERT has seen 2M extra gradient updates and 2M extra constraint batches compared to BERT. Without a BERT baseline trained for the same number of total updates, or a control auxiliary task with shuffled labels, you cannot attribute the improvements to the lexical content of the constraints.\n\nThe paper does other things well. The architecture is simple and clearly described; the fine-grained analysis on the diagnostic dataset is a nice touch; and the lexical simplification results are consistent across three datasets. The authors are honest about training from scratch in identical conditions for the MLM part, and they do not claim state-of-the-art absolute numbers.\n\nThe soft spots are significant. Besides the confound, they report single runs with no variance or significance testing; many GLUE deltas are tiny (+0.2 on QQP, +0.3 on MNLI), while the big gains on CoLA and AX are exactly where seed sensitivity can be large. They also promise code but give no URL. So the reader's conditional verdict is fair, and the stress-test note is on target: the missing control is the load-bearing flaw.\n\nWho is this for? Someone working on knowledge-enhanced pretraining, as a proof-of-concept that lexical constraints can be plugged into BERT without breaking it. It does not establish a benefit.\n\nMy recommendation: send it to peer review, but expect heavy revision. The authors need to add matched-compute baselines and variability estimates. If they cannot reproduce the effect under a proper control, the paper should not be accepted as is.\n\nBest,","headline":"A promising idea undermined by a missing control: LIBERT's gains over BERT may be due to extra optimizer steps and data, not lexical knowledge.","tokens_in":14566,"tokens_out":2953,"would_cite":false,"duration_ms":31930,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a lexical-relation classification task to BERT pretraining yields a model that beats standard BERT on 9 of 10 GLUE tasks and improves lexical simplification by up to 8.2 points.","keywords":["BERT","multi-task pretraining","lexical relation classification","semantic similarity","lexical simplification","GLUE benchmark","WordNet","distributional semantics"],"falsifier":"Train LIBERT with the same architecture and data but randomize the similarity labels of the constraint pairs; if its GLUE and lexical-simplification scores still beat BERT, the reported gains are not caused by lexical content. A complementary check is to hold out a random subset of WordNet synonym pairs from pretraining and test the LRC head on them: near-chance accuracy on held-out pairs would indicate the classifier did not learn a transferable notion of similarity.","tokens_in":13638,"feed_emoji":"📚","tokens_out":11967,"duration_ms":107613,"temperature":0.7,"pith_summary":"Unsupervised language models such as BERT learn word meaning from text co-occurrence alone, which makes it hard for them to separate genuine similarity (car/automobile) from broader topical relatedness (car/road). This paper asks whether that gap can be closed by adding an explicit pretraining task that classifies word pairs as semantically similar or not, using synonym and hypernymy pairs from WordNet and Roget's Thesaurus. The authors report that the resulting model, LIBERT, beats a same-configuration BERT on 9 of 10 GLUE test tasks after 2M pretraining steps (and ties on QNLI), with the largest gains on linguistic acceptability and the diagnostic AX set, and improves lexical simplification on all three evaluated benchmarks, up to 8.2 accuracy points. If true, the result matters because it offers a low-cost route to inject curated lexical knowledge into large pretrained encoders, and because it shows that distributional pretraining leaves a measurable, correctable deficit in word-level similarity.","feed_headline":"WordNet-trained BERT variant beats BERT on 9 of 10 GLUE tasks","feed_subtitle":"Adding lexical relation classification to pretraining also lifts lexical simplification by up to 8.2 points.","key_machinery":"The load-bearing component is the Lexical Relation Classification (LRC) objective: a two-way softmax classifier on the [CLS] representation of a BERT-compatible word-pair input that decides whether the two words stand in a true semantic-similarity relation. Positive examples are synonym and direct hypernym/hyponym pairs from WordNet and Roget; negative examples are built by taking, for each positive pair, the words from the same mini-batch that are closest in a fastText space, so the classifier must learn to separate genuine similarity from near-miss relatedness. LRC's gradient is applied to the transformer in an alternating regime with MLM and NSP batches, which is the mechanism by which lexical knowledge is written into the contextualized representations.","core_discovery":"The paper's central claim is that a binary lexical-relation classification head, trained alongside BERT's masked language modeling and next sentence prediction on 1.35M synonym and direct hypernym/hyponym pairs, steers contextualized representations toward true word-level semantic similarity. With all other hyperparameters and pretraining data kept identical, the resulting LIBERT model outperforms standard BERT on 9 of 10 GLUE test tasks after 2M steps and matches it on QNLI; the biggest margins appear on CoLA, AX, and SST-2, and the advantage persists when pretraining is extended from 1M to 2M steps. On lexical simplification, LIBERT improves both candidate generation and the full replacement pipeline on all three datasets, with an accuracy gain of up to 8.2 points on LexMTurk. The authors read these results as evidence that clean external lexical knowledge complements, rather than duplicates, the distributional signal in large-scale language modeling.","pith_inferences":["The same auxiliary-task recipe could be applied to other lexical relations (antonymy, meronymy, lexical entailment) and to other pretrained encoders; the paper tests only synonymy and hypernymy in English, so this is an extension, not a result.","The diagnostic gains in Factivity and Morphological Negation suggest the injected constraints sharpen the model's compositionality, so a targeted test would be to measure LIBERT on morphological generalization or lexical entailment datasets rather than broad GLUE tasks.","If the LRC signal is truly complementary, training both models much longer (or on more text) should preserve at least part of the gap; a direct test would extend their pretraining beyond 2M steps and track deltas.","The negative-sampling choice is a likely active ingredient: replacing fastText-based near-miss negatives with random non-synonym pairs might shrink or erase the gains, which would indicate that contrastive difficulty, not just pair labels, drives the specialization."],"forward_implications":["On the GLUE test sets after 2M pretraining steps, LIBERT beats BERT on 9 of 10 tasks and matches it on QNLI; the largest deltas are on CoLA (+6.5), AX (+2.5), and SST-2 (+1.1).","On the GLUE diagnostic set, LIBERT's largest improvements are in Lexical Semantics (up to +29.7 at 2M), with large gains also in Logic and Knowledge & Common Sense at 1M, suggesting lexical knowledge partially supports factual and logical inference.","In lexical simplification, LIBERT improves both candidate generation (F1 gains of +1.3 to +3.0 points at 2M across datasets) and full-pipeline accuracy, with the largest gain +8.2 points on LexMTurk.","The advantage of lexical injection does not disappear as pretraining continues: deltas over BERT after 2M MLM+NSP steps are comparable to or larger than after 1M.","Because LIBERT and BERT are trained from scratch under identical settings, the reported differences isolate the effect of the LRC task rather than other pretraining variables."],"supporting_citations":[{"why":"Defines BERT, its pretraining corpus, and the MLM/NSP objectives; LIBERT is BERT with LRC added, and BERT is the comparison baseline.","marker":"Devlin et al., 2019"},{"why":"Supplies WordNet, the source of the synonym and hypernymy constraints used to train the lexical relation classifier.","marker":"Miller, 1995"},{"why":"Provides the 326,187 direct hyponym–hypernym pairs from WordNet used as positive LRC constraints.","marker":"Vulić and Mrkšić, 2018"},{"why":"Establishes the practice of constructing negatives from nearest words in an auxiliary space, which LRC uses to create non-similar pairs.","marker":"Wieting et al., 2015"},{"why":"FastText supplies the auxiliary embedding space for picking near-miss negatives and serves as a ranking feature in lexical simplification.","marker":"Bojanowski et al., 2017"},{"why":"Defines the GLUE benchmark, its development/test splits, and the diagnostic set used in the linguistic analysis.","marker":"Wang et al., 2018"},{"why":"Provides the BERT-LS method that the paper adapts for candidate generation and ranking of lexical simplification substitutes.","marker":"Qiang et al., 2019"},{"why":"Supplies the LexMTurk dataset, one of the three lexical simplification benchmarks, and the accuracy metric used for the full pipeline.","marker":"Horn et al., 2014"},{"why":"Supplies the BenchLS dataset, a merged lexical simplification benchmark used in the evaluation.","marker":"Paetzold and Specia, 2016"},{"why":"Supplies the NNSeval dataset for non-native-speaker lexical simplification and frames the task's candidate-generation and ranking split.","marker":"Paetzold and Specia, 2017"}],"fun_headline_variants":["WordNet-enhanced BERT beats vanilla on 9/10 GLUE tasks","LIBERT: WordNet-aware BERT outperforms on 9/10 GLUE benchmarks","Adding WordNet knowledge to BERT improves GLUE and simplification","Lexically informed BERT surpasses baseline on 9/10 GLUE tasks","BERT with WordNet relations beats original on 9/10 GLUE tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the LRC task teaching a generalizable sense of similarity rather than memorizing the 1.35M training pairs or their lexical form.","fun_headline_variants_meta":{"raw":{"variants":["WordNet-enhanced BERT beats vanilla on 9/10 GLUE tasks","LIBERT: WordNet-aware BERT outperforms on 9/10 GLUE benchmarks","Adding WordNet knowledge to BERT improves GLUE and simplification","Lexically informed BERT surpasses baseline on 9/10 GLUE tasks","BERT with WordNet relations beats original on 9/10 GLUE tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001104,"raw_usage":{"total_tokens":4606,"prompt_tokens":953,"completion_tokens":3653,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":3551}},"tokens_in":569,"tokens_out":3653,"duration_ms":27053,"temperature":1.0,"reasoning_tokens":3551,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:52:40.228225+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train LIBERT with the same architecture and data but randomize the similarity labels of the constraint pairs; if its GLUE and lexical-simplification scores still beat BERT, the reported gains are not caused by lexical content. A complementary check is to hold out a random subset of WordNet synonym pairs from pretraining and test the LRC head on them: near-chance accuracy on held-out pairs would indicate the classifier did not learn a transferable notion of similarity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines BERT, its pretraining corpus, and the MLM/NSP objectives; LIBERT is BERT with LRC added, and BERT is the comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the practice of constructing negatives from nearest words in an auxiliary space, which LRC uses to create non-similar pairs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GLUE benchmark, its development/test splits, and the diagnostic set used in the linguistic analysis."},{"cited_title":"Lexical Simplification with Pretrained Encoders","cited_arxiv_id":"1907.06226","evidence_quote":"Provides the BERT-LS method that the paper adapts for candidate generation and ranking of lexical simplification substitutes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BenchLS dataset, a merged lexical simplification benchmark used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NNSeval dataset for non-native-speaker lexical simplification and frames the task's candidate-generation and ranking split."}],"review_version":1}