A deterministic n-gram benchmark for Japanese open-ended QA, built from LLM-generated reference answer sets, that reports a 0.9896 correlation with GPT-4o judge scores.
Are LLMs Models of Distributional Semantics? A Case Study on Quantifiers
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Distributional semantics is the linguistic theory that a word's meaning can be derived from its distribution in natural language (i.e., its use). Language models are commonly viewed as an implementation of distributional semantics, as they are optimized to capture the statistical features of natural language. It is often argued that distributional semantics models should excel at capturing graded/vague meaning based on linguistic conventions, but struggle with truth-conditional reasoning and symbolic processing. We evaluate this claim with a case study on vague (e.g. "many") and exact (e.g. "more than half") quantifiers. Contrary to expectations, we find that, across a broad range of models of various types, LLMs align more closely with human judgements on exact quantifiers versus vague ones. These findings call for a re-evaluation of the assumptions underpinning what distributional semantics models are, as well as what they can capture.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
A Judge-free LLM Open-ended Generation Benchmark Based on the Distributional Hypothesis
A deterministic n-gram benchmark for Japanese open-ended QA, built from LLM-generated reference answer sets, that reports a 0.9896 correlation with GPT-4o judge scores.