Pith. sign in

REVIEW 3 cited by

Language models are not naysayers: An analysis of language models on negation benchmarks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2306.08189 v1 pith:E3RI5LYP submitted 2023-06-14 cs.CL

Thinh Hung Truong , Timothy Baldwin , Karin Verspoor , Trevor Cohn This is my paper
classification cs.CL
keywords negationlanguagellmsmodelsbeenbenchmarksevaluateincluding
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Negation has been shown to be a major bottleneck for masked language models, such as BERT. However, whether this finding still holds for larger-sized auto-regressive language models (``LLMs'') has not been studied comprehensively. With the ever-increasing volume of research and applications of LLMs, we take a step back to evaluate the ability of current-generation LLMs to handle negation, a fundamental linguistic phenomenon that is central to language understanding. We evaluate different LLMs -- including the open-source GPT-neo, GPT-3, and InstructGPT -- against a wide range of negation benchmarks. Through systematic experimentation with varying model sizes and prompts, we show that LLMs have several limitations including insensitivity to the presence of negation, an inability to capture the lexical semantics of negation, and a failure to reason under negation.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TNG-CLIP:Training-Time Negation Data Generation for Negation Awareness of CLIP

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A training-time pipeline that synthesizes diverse negation captions from batch neighbors improves CLIP's negation accuracy on matching and generation benchmarks, and a new NEG-TTOI benchmark measures negation handling...

  2. QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A contextual bandit that chooses among five query-rewrite strategies, conditioned on 17 linguistic features, reduces LLM hallucination on QA benchmarks and beats static prompting and no-rewrite baselines.

  3. Framing Instability in LLM Ethical Stance: Auditing Negation Sensitivity in Moral Dilemmas

    cs.AI 2026-01 conditional novelty 4.0 of 10

    Small open-weight LLMs endorse prohibited actions 24% of the time under affirmative framing but 77-100% under negated framings, a polarity swing that threatens high-stakes AI deployment.

Pith tools