{"id":"d573414f-7dd7-461b-a46e-030ac6a9567f","arxiv_id":"2502.07717","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Further pre-training BERT and RoBERTa on next-sentence polarity prediction and a polarity-reversing variant of next sentence prediction improves negation reasoning by 1.8 to 9.3 accuracy points on CondaQA.","lead":"The paper introduces two self-supervised pre-training tasks that teach BERT and RoBERTa to pay attention to negation, and reports consistent gains on nine benchmarks. A generalist might care because small, cheap language models are notoriously bad at understanding 'not', and this is a simple, unlabeled recipe to improve them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The further pre-training corpus is drawn from 2024 Wikipedia without deduplication against CondaQA, whose passages also come from Wikipedia; the headline CondaQA gains may reflect train/eval overlap rather than general negation robustness.","rationale":"Good-faith reading: the authors propose two self-supervised objectives and show consistent gains across many negation benchmarks. The reversal-rule validation (96% of 100 samples) and the use of only not/n't/never are real limitations, but they are acknowledged, and the transfer to 200+ cues is empirically plausible. The load-bearing issue I keep coming back to is not whether the generated negatives are semantically perfect; it is whether the evaluation itself is contaminated by the pre-training corpus. Section 3.3 draws from Wikipedia; CondaQA is also extracted from Wikipedia (Section 4.1). Nothing in the paper describes removing benchmark source passages from the pre-training set. This is especially acute because the further pre-training happens on a 2024 Wikipedia dump, while CondaQA was released in 2022; the exact source passages are likely present. The paper's own limitation section discusses cue coverage and English-only data, but never mentions overlap. The NLI and NLU gains are less likely to be affected, but the headline number (CondaQA group consistency 2.4->17.6%) is the strongest evidence for the central claim and is exactly the result that contamination would inflate. I therefore do not regard the reversal-rule artifact worry as the most load-bearing; even if reversal were perfect, overlap would still threaten the claim. The check I propose is concrete and could be run from the released code/data. If overlap is zero, the concern is refuted and the conditional verdict stands. If overlap is material, the paper would need to show the gains survive deduplication.","tokens_in":25589,"tokens_out":6489,"duration_ms":58326,"concrete_test":"Compute exact and near-duplicate sentence overlap between the released pre-training corpus (or regenerate it with the released script) and the 1,289 CondaQA source passages and all edited passages. Then retrain the most prominent model (RoBERTa-base + NSP, which gives 2.4%->17.6% group consistency) on the same 1M-pair corpus after removing any overlapping S1/S2 pairs, and re-evaluate CondaQA with identical fine-tuning hyperparameters. If the gain mostly disappears, the central claim is confounded; if the gain persists, overlap is not the explanation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most influential result is the CondaQA improvement, including RoBERTa-base group consistency rising from 2.4% to 15.6-17.6% (Table 1). For this to be evidence of general negation robustness, the pre-training signal must not contain the evaluation passages. Section 3.3 builds the 6.4M-pair corpus by extracting 'all sentences from Wikipedia' containing negation, using the 2024 Wikimedia dump. Section 4.1 states CondaQA was created by extracting 1,289 passages from English Wikipedia that contained negation phrases. No statement in the paper, Limitations, or appendices rules out overlap: there is no deduplication of the pre-training data against CondaQA or any downstream benchmark. Because the further pre-training is on a Wikipedia snapshot that postdates CondaQA's construction, the exact original passages (and likely their surrounding sentence pairs) can appear in the NSPP/NSP training data. Off-the-shelf BERT/RoBERTa were also pre-trained on Wikipedia, but on an older snapshot; the additional 500K-1M steps on the modern dump can add direct exposure to the evaluation passages. If the headlining CondaQA gains partly reflect exposure to the same texts, the central claim of robustness via polarity-reversal pre-training is weakened, regardless of the reversal rules' 96% accuracy. This is testable by measuring overlap and retraining on a deduplicated corpus.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two self-supervised further pre-training tasks, Next Sentence Polarity Prediction (NSPP) and a polarity-reversed variant of Next Sentence Prediction (NSP), and applies them to BERT and RoBERTa base/large models. The authors construct approximately 6.4M (S1, S2) sentence pairs from a 2024 Wikipedia dump using rule-based polarity reversal, further pre-train on subsets of up to 1M pairs, and evaluate on CondaQA and eight other negation-related benchmarks. They report statistically significant accuracy gains on CondaQA for all model configurations, with RoBERTa-base group consistency rising from 2.4% to 15.6-17.6%, and also report improvements on NLI, NLU, and LAMA-Neg benchmarks. The paper's central claim is that this self-supervised recipe provides general robustness against negation.","tokens_in":25930,"tokens_out":6704,"duration_ms":55829,"significance":"If the gains are real and not an artifact of benchmark overlap, this is a practical contribution: it offers a simple, self-supervised, architecture-agnostic way to improve negation handling in smaller language models, with publicly released code and data. The use of McNemar's test for the main CondaQA accuracy comparisons is a strength relative to much prior work that reports only point estimates. However, the headline CondaQA result is vulnerable to train/eval overlap with the 2024 Wikipedia snapshot, and the LAMA degradation of the jointly pre-trained RoBERTa models conflicts with the paper's no-harm claim on non-negated inputs. The significance of the contribution therefore hinges on the outcome of a deduplication analysis.","major_comments":[{"comment":"The pre-training corpus is built from a 2024 Wikipedia dump (Section 3.3), and CondaQA passages are also extracted from English Wikipedia (Section 4.1), but the paper never states that the pre-training data were deduplicated against CondaQA or any other downstream benchmark. Because the snapshot postdates CondaQA's construction, exact CondaQA passages and their neighboring sentences can appear in the NSPP/NSP training pairs, especially since the method extracts all sentences containing negation together with their preceding sentences. This makes it impossible to attribute the reported CondaQA gains (Table 1) to general negation robustness rather than to direct exposure to the evaluation text. Please report overlap statistics (e.g., exact sentence matches, high-n-gram overlaps) between the pre-training corpus and the CondaQA passages, and ideally rerun the main CondaQA evaluation using a deduplicated corpus.","section":"§3.3, §4.1"},{"comment":"The text states that \"the fact that the models remain within ±1.65% of the vanilla models\" demonstrates maintenance of performance on non-negated input, but this is contradicted by the RoBERTa joint models in Table 5: RoBERTa-base + NSPP+NSP drops from 14.77 to 12.06 on ConceptNet and from 11.93 to 6.76 on TREx, and RoBERTa-large + NSPP+NSP drops from 15.48 to 3.68 on TREx. The sentence acknowledges \"except for RoBERTa models jointly pre-trained\" but the conclusion does not incorporate the magnitude of this exception. Additionally, Appendix F states that EWC regularization is applied only for the LAMA/LAMA-Neg evaluations, so those models are pre-trained with a different objective; please clarify whether the LAMA results come from a separate pre-training run and whether the no-harm claim is intended to be limited to that configuration.","section":"§6.2, Table 5"},{"comment":"When constructing the affirmative sentences, the authors \"verify that none of the negation cues from CondaQA are present.\" This imports the evaluation benchmark's cue list into the pre-training data construction. The choice is not motivated in the paper, and it means the pre-training distribution is shaped by one specific downstream test. Please justify this choice or show that the results are insensitive to using a generic, benchmark-independent negation cue list.","section":"§3.3"}],"minor_comments":[{"comment":"The text reports \"1.8%-9.3% higher accuracy\" while the abstract and Table 1 support a maximum of 9.1% (RoBERTa-base, 48.2 to 57.3); please correct the inconsistency.","section":"§6.1"},{"comment":"The header contains a typo: \"BEBT-base\" should be \"BERT-base\".","section":"Table 8"},{"comment":"It would help to state explicitly that the LAMA/LAMA-Neg models are pre-trained with EWC and therefore are not the same checkpoints as the CondaQA/NLI/NLU models, so results across tables are not directly comparable.","section":"Appendix F"},{"comment":"For BERT-base, NSPP and NSPP+NSP group consistency \"All\" (2.7% and 2.9%) are lower than the vanilla 3.4%; the text calls this \"marginal,\" but it is a decrease, so please phrase the exception more precisely.","section":"Table 1"},{"comment":"The QNLI example label \"Not Entailment (i.e., question is not answered)\" is confusing because the parenthetical gloss does not fully capture the entailment distinction; consider clarifying the label.","section":"Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The overlap issue is the most serious concern. Even if exact CondaQA passages do not appear in the pre-training data, high n-gram overlap with the 2024 Wikipedia snapshot could inflate the headline gains. A deduplication analysis and, ideally, an evaluation on a deduplicated corpus is the cleanest way to resolve this. The LAMA exception for the joint RoBERTa models should also be presented prominently rather than in a parenthetical, because it weakens the claim that the method does not harm non-negated inputs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, simple further-pretraining paper that probably works, but the headline CondaQA numbers have a train/eval overlap problem that the authors need to rule out before I'd trust them at face value.\n\nThe new stuff is genuinely new: NSPP (predict whether the next sentence contains negation) and the add/remove variant of NSP are a clean extension of Singh et al., who only add negation. The recipe is unlabeled, easy to reproduce, and the gains hold across four model sizes and nine benchmarks, with McNemar significance. The ablation showing that polarity reversal beats only-add or only-remove is the right experiment, and it supports their mechanism. They also ship code and data, which makes this a useful baseline for the negation-robustness line.\n\nThe soft spots are real but manageable. The biggest one is the Wikipedia overlap. Section 3.3 builds the pre-training corpus from the 2024 Wikipedia dump; CondaQA was also built from Wikipedia passages. The paper never mentions deduplication. If the exact CondaQA passages (or their neighboring sentences) appear in the pre-training data, then the 2.4% to 15.6-17.6% jump for RoBERTa-base on group consistency could be partly memorization, not general robustness. This is testable: measure overlap, retrain on a deduplicated corpus. The authors should do that. It doesn't kill the paper, because the gains on LAMA-neg and the negated NLI pairs are independent of CondaQA, but it does mean the headline number is suspect until checked.\n\nSecond, all results are single-run point estimates. No seeds, no confidence intervals. For a paper whose main evidence is small-to-moderate accuracy differences, that's a weakness. It's a fast fix, but it should be a revision condition.\n\nThe 100-sample manual validation of the reversal rules is a minor concern. 96% correctness on the sampled sentences is fine for data generation; a 4% error rate just injects noise, and the transfer to CondaQA's 200+ cues suggests the model isn't memorizing three cues. The grammatical awkwardness of some reversed sentences (e.g., 'She might have not been sleeping') could be a surface artifact, but the NLI and LAMA-neg results make me think the effect is mostly general.\n\nWho's this for? Anyone working on negation robustness, QA, or fact-checking with small deployed models. It's worth a serious referee, but the referee should ask for the overlap analysis and multi-seed numbers before publication.","headline":"Solid, simple negation pre-training with real gains, but the CondaQA headline numbers need a dedup check before I'd trust them.","tokens_in":26424,"tokens_out":2511,"would_cite":true,"duration_ms":22061,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Further pre-training on two self-supervised polarity tasks makes BERT and RoBERTa substantially better at answering questions that hinge on negation.","keywords":["negation","language model pre-training","next sentence prediction","polarity reversal","self-supervised learning","BERT","RoBERTa","CondaQA"],"falsifier":"Train the same models on polarity-reversed pairs in which the reversed sentence has been hand-edited or generated by a grammar-aware system to read naturally, and also on randomly selected negative pairs matched for length and lexical overlap; if the CondaQA gains persist only with the rule-based reversed pairs, or if the gains concentrate on CondaQA items whose cues overlap with not, n't, and never, the claim that the model has learned general negation robustness is undercut.","tokens_in":25363,"feed_emoji":"🚫","tokens_out":6930,"duration_ms":56413,"temperature":0.7,"pith_summary":"The paper tries to establish that language models' well-known blindness to negation can be reduced by a cheap, fully self-supervised pre-training step that forces the model to attend to whether a sentence is negated. The authors introduce two training tasks built from unlabeled Wikipedia text: Next Sentence Polarity Prediction (NSPP), in which the model sees only the first sentence and must predict whether the next sentence contains a negation cue, and a modified Next Sentence Prediction (NSP) in which the negative example is the actual next sentence with its polarity reversed rather than a random sentence. Pre-training BERT and RoBERTa on either task consistently outperforms the off-the-shelf models on nine negation-related benchmarks, with the clearest gains on CondaQA, a question-answering corpus built around negation: accuracy rises 1.8%-9.1% and RoBERTa-base's group consistency jumps from 2.4% to 15.6%-17.6%. If this is right, robustness to negation can be instilled without new annotation, extra data beyond ordinary text, or any change at prediction time.","feed_headline":"Polarity-flip pre-training lifts negation QA by up to 9.1%","feed_subtitle":"RoBERTa-base's group consistency on CondaQA jumps from 2.4% to 17.6% after pre-training on polarity-reversed next sentences.","key_machinery":"The load-bearing mechanism is automatic polarity reversal with hand-built rules. The rules operate only on sentences whose negation cue is not, n't, or never, where the cue modifies the main verb, the sentence is not a question, and exactly one cue is present; adding negation inserts or attaches the cue to the auxiliary or main verb and swaps polarity items (some to any, already to yet), while removing negation deletes the cue, restores the auxiliary's lemma, and swaps back (any to some, yet to already, but to and). This produces the 6.4M training pairs for the two tasks. The NSP variant is the key change: negative examples are true next sentences whose polarity has been flipped, so the model must treat polarity reversal as a coherence violation rather than random unrelatedness.","core_discovery":"The paper's central claim is that further pre-training on two polarity-focused self-supervised tasks makes BERT and RoBERTa genuinely more robust to negation, not just better on individual datasets. Given only a sentence, the model must predict whether the following sentence will contain negation (NSPP); and in a redesigned NSP, a proposed continuation is judged incoherent when its polarity has been reversed by adding or removing a negation cue, replacing the random-negative sampling of the original task. On CondaQA, all pre-trained variants show statistically significant accuracy gains and large gains in group consistency, the metric that requires answering correctly across original, paraphrased, scope-changed, and affirmed versions of the same passage. The authors interpret this as evidence that the model has learned something about how negation alters meaning and discourse coherence, and that this transfers to a corpus whose 200+ negation cues go well beyond the three cues used to create the training data.","pith_inferences":["The manual reversal rules were validated on only 100 sentences, with 96% judged correct; if the 4% errors and grammatically awkward outputs such as 'She might have not been sleeping' create surface regularities, part of the CondaQA gain could come from learning those artifacts rather than negation semantics. A control study with human-edited reversed sentences would separate these.","Because NSPP's signal is weaker and joint training sometimes hurts, the two tasks may be teaching overlapping but partly competing skills: detecting a cue's presence versus judging a discourse-coherence violation. This suggests the coherence-based NSP is the more promising route for further work.","The method's restriction to not, n't, and never leaves open whether a richer set of cues in pre-training would extend the gains to negation types absent from CondaQA, or to languages with different negation morphology; the authors note the tasks are language-agnostic but only test English.","Pre-training on polarity-reversed sentences might serve as a general-purpose robustness intervention for other semantic phenomena with a similar duality, such as tense or modality, where flipping a sentence's value creates a natural negative training signal."],"forward_implications":["Further pre-training on NSP gives the largest and most consistent gains; it outperforms pre-training on NSPP and, except for RoBERTa-large, joint training does not add benefit.","The gains transfer beyond the three cues used in training: CondaQA contains over 200 negation cue types, including affixal and multi-word cues, yet accuracy improves there most.","Models pre-trained on polarity reversal outperform prior negation-specific approaches, including affirmative-interpretation augmentation and unlikelihood or negation-augmented pre-training, without extra inference-time computation.","On LAMA-Neg, error rates drop by 0.09 to 19.96 points, while on original LAMA and NLI/NLU datasets without negation, performance stays within roughly plus or minus 1.65%, indicating the pre-training does not trade away general ability.","Ablations show that mixing added and removed negation, that is true polarity reversal, beats training on either direction alone, and adding negation is a stronger training signal than removing it."],"supporting_citations":[{"why":"Supplies BERT and the original Next Sentence Prediction task whose random-negative sampling this paper replaces with polarity reversal.","marker":"Devlin et al. (2019)"},{"why":"Supplies the RoBERTa model family and pre-training setup that the further pre-training continues.","marker":"Liu et al. (2019)"},{"why":"Defines CondaQA, the main benchmark, including the group-consistency metric and the artifact checks the paper relies on.","marker":"Ravichander et al. (2022)"},{"why":"Contributes the negated RTE, SNLI, and MNLI evaluation pairs used as additional benchmarks.","marker":"Hossain et al. (2020)"},{"why":"Provides LAMA-Neg, the negated probe whose error rates drop after pre-training.","marker":"Kassner and Schütze (2020)"},{"why":"Provides the negation-aware analysis and QNLI, WiC, and WSC benchmark formats used for NLU evaluation.","marker":"Hossain et al. (2022)"},{"why":"Gives the affirmative-interpretation baseline the paper compares against and outperforms on CondaQA.","marker":"Rezaei and Blanco (2024)"},{"why":"Prior negation-augmented pre-training (NBERT and NRoBERTa) used as a baseline for comparison.","marker":"Singh et al. (2023)"},{"why":"Prior unlikelihood-style negation training (BERTNOT) used as a baseline for comparison.","marker":"Hosseini et al. (2021)"},{"why":"Supplies the English Wikipedia corpus from which all roughly 6.4M pre-training sentence pairs are extracted.","marker":"Wikimedia-Foundation (2024)"}],"fun_headline_variants":["Self-supervised polarity tasks boost negation robustness by 9.1%","New pre-training tasks make BERT and RoBERTa negation-proof","Polarity pre-training lifts negation QA accuracy up to 9.1%","Teaching models polarity flips yields large negation QA gains","Next-sentence polarity prediction sharpens negation handling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that automatically reversing polarity in Wikipedia sentences, using rules validated on 100 examples and limited to not, n't, and never, produces training examples whose only meaningful difference is the presence or absence of negation, with no systematic grammatical or stylistic artifact for the model to latch onto.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised polarity tasks boost negation robustness by 9.1%","New pre-training tasks make BERT and RoBERTa negation-proof","Polarity pre-training lifts negation QA accuracy up to 9.1%","Teaching models polarity flips yields large negation QA gains","Next-sentence polarity prediction sharpens negation handling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1301,"prompt_tokens":856,"completion_tokens":445,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":357}},"tokens_in":472,"tokens_out":445,"duration_ms":5155,"temperature":1.0,"reasoning_tokens":357,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T11:46:51.374164+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same models on polarity-reversed pairs in which the reversed sentence has been hand-edited or generated by a grammar-aware system to read naturally, and also on randomly selected negative pairs matched for length and lexical overlap; if the CondaQA gains persist only with the rule-based reversed pairs, or if the gains concentrate on CondaQA items whose cues overlap with not, n't, and never, the claim that the model has learned general negation robustness is undercut.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the English Wikipedia corpus from which all roughly 6.4M pre-training sentence pairs are extracted."}],"review_version":1}