FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large Language Models

Pengfei Liu; Shichao Sun; Yiyuan Li

RECORD

FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large Language Models

Yiyuan Li, Shichao Sun, Pengfei Liu

Artificial Intelligence · Submitted 1 Jul 2024

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Our experimental findings reveal that fuzzy reasoning continues to pose significant challenges for LLMs.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2407.01046 (version 2)

Pith IDUM7TDEU3, Pith's permanent record of this paper and its review

CitationPengfei Liu, Shichao Sun, Yiyuan Li. "FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large Language Models." arXiv:2407.01046 (2024). https://pith.science/paper/UM7TDEU3

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,414 input tokens.

What the abstract itself states, as probabilities: claims first 0.17 · states limitation 0.32 · human participants 0.06 · controlled comparison 0.21 · randomized assignment 0.02 · states quantitative result 0.06 · compares with prior results 0.77 · mentions code or data availability 0.04.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; show every role.

Fuzzy reasoning is vital due to the frequent use of imprecise information in daily contexts. However, the ability of current large language models (LLMs) to handle such reasoning remains largely uncharted. In this paper, we introduce a new benchmark, FRoG, for fuzzy reasoning, featuring real-world mathematical word problems that incorporate generalized quantifiers. Our experimental findings reveal that fuzzy reasoning continues to pose significant challenges for LLMs. Moreover, we find that existing methods designed to enhance reasoning do not consistently improve performance in tasks involving fuzzy logic. Additionally, our results show an inverse scaling effect in the performance of LLMs on FRoG. Interestingly, we also demonstrate that strong mathematical reasoning skills are not necessarily indicative of success on our benchmark.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Fuzzy reasoning is vital due to the frequent use of imprecise information in daily contexts.	background 1.00	0.04	0.02	asserted 1.00
s1	However, the ability of current large language models (LLMs) to handle such reasoning remains largely uncharted.	gap 1.00	0.07	0.03	asserted 0.96
s2	In this paper, we introduce a new benchmark, FRoG, for fuzzy reasoning, featuring real-world mathematical word problems that incorporate generalized quantifiers.	contribution 0.99	0.89	0.05	asserted 1.00
s3	Our experimental findings reveal that fuzzy reasoning continues to pose significant challenges for LLMs.	result 1.00	0.92	0.53	asserted 1.00
s4	Moreover, we find that existing methods designed to enhance reasoning do not consistently improve performance in tasks involving fuzzy logic.	result 0.99	0.93	0.15	asserted 0.99
s5	Additionally, our results show an inverse scaling effect in the performance of LLMs on FRoG.	result 1.00	0.96	0.12	asserted 1.00
s6	Interestingly, we also demonstrate that strong mathematical reasoning skills are not necessarily indicative of success on our benchmark.	result 0.95	0.96	0.11	asserted 1.00

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.99.

What was examined, and what was not

sources read

arXiv listing abstract v2

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Fuzzy reasoning is vital due to the frequent use of imprecise information in daily contexts. However, the ability of current large language models (LLMs) to handle such reasoning remains largely uncharted. In this paper, we introduce a new benchmark, FRoG, for fuzzy reasoning, featuring real-world mathematical word problems that incorporate generalized quantifiers. Our experimental findings reveal that fuzzy reasoning continues to pose significant challenges for LLMs. Moreover, we find that existing methods designed to enhance reasoning do not consistently improve performance in tasks involving fuzzy logic. Additionally, our results show an inverse scaling effect in the performance of LLMs on FRoG. Interestingly, we also demonstrate that strong mathematical reasoning skills are not necessarily indicative of success on our benchmark.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 30 sentences without a confident label

Uses the method or tool of 5 papers

We evaluated several open-sourced LLMs, including Llama-2 [ref] , CodeLlama [ref] , Llemma [ref] , llama-3 [ref] , Mistral [ref] , Qwen-1.5 [ref] , Tulu-2 [ref] , WizardLM [ref] , WizardMath [ref] and Yi-Chat [ref] on our benchmark.

WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct · Experiment · 0.81
We evaluated several open-sourced LLMs, including Llama-2 [ref] , CodeLlama [ref] , Llemma [ref] , llama-3 [ref] , Mistral [ref] , Qwen-1.5 [ref] , Tulu-2 [ref] , WizardLM [ref] , WizardMath [ref] and Yi-Chat [ref] on our benchmark.

Code Llama: Open Foundation Models for Code · Experiment · 0.81
We evaluated several open-sourced LLMs, including Llama-2 [ref] , CodeLlama [ref] , Llemma [ref] , llama-3 [ref] , Mistral [ref] , Qwen-1.5 [ref] , Tulu-2 [ref] , WizardLM [ref] , WizardMath [ref] and Yi-Chat [ref] on our benchmark.

Mistral 7B · Experiment · 0.81

Uses data or values from 1 paper

To this end, we have developed , a benchmark for Fuzzy Reasoning benchmark of Generalized quantifiers, which transforms real-world mathematical problems from GSM8K [ref] and MathQA [ref] datasets into multiple-choice questions.

Training Verifiers to Solve Math Word Problems · Introduction · 0.93

Cites as background 5 papers

Scaling law is introduced in [ref] to suggest the phenomenon that LLMs can achieve enhanced task performance by scaling up model sizes.

Scaling Laws for Neural Language Models · Q2: Can the scaling law be observed in ? · 1.00
Existing works like [ref] investigate employing fuzzy probabilistic theory in association-based problems, like the relation between fuzzy features and labels (e.g. a lot of smoking and cancer).

A Fundamental Probabilistic Fuzzy Logic Framework Suitable for Causal Reasoning · Introduction · 0.84
Various approaches have been developed to enhance the math reasoning ability of LLMs, like prompt-based designs [ref] .

Progressive-Hint Prompting Improves Reasoning in Large Language Models · Related Work · 0.85

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.